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To  the  Reader 


Abstract  The  purpose  of  this  document  is  to  summarize  the  major  elements  of  the 

Systems  Engineering  Capability  Maturity  Model  (SE-CMM^ )  appraisal 
method  (SAM).  SAM  is  a  method  for  using  the  SE-CMM  to 
benchmark,  or  otherwise  appraise,  the  process  capability  of  an 
organization's  or  enterprise's  systems  engineering  function.  The  SE- 
CMM  itself  is  described  in  SECMM-94-04ICMU/SEI-96-HB-004 
[SECMM].  This  document  describes  each  step  of  an  SE-CMM 
appraisal  and  provides  guidance  for  the  preparation  and  conduct  of  an 
appraisal.  It  also  contains  background  and  context  information  about 
the  appraisal  method. 


Who  should  use  Organizations,  enterprises,  or  projects  performing  significant  systems 
SAM?  engineering  activities  are  candidates  for  using  SAM.  Organizations 

involved  in  systems  engineering  that  need  to  understand  their  use  and 
management  of  common  systems  engineering  practice  can  use  SAM  as 
the  starting  point  for  an  effort  to  improve  their  systems  engineering 
process. 


Why  was  it  SAM  was  developed  to  provide  the  systems  engineering  community 

developed?  with  a  publicly-accessible  method  for  preparing  for  and  performing  SE- 

CMM  appraisals. 


What  is  the  Although  the  basic  concepts  in  SAM  are  adaptable  to  most 

scope  of  SAM?  organizational  appraisal  contexts,  the  scope  of  SAM  is  designed 

specifically  to  support  the  SE-CMM.  The  activity  set  for  SAM  is  the 
same  basic  set  as  that  used  by  the  SEI  CMM-Based  Appraisal  for 
Internal  Process  Improvement  (CBA-IPI)  method;  however,  differences 
from  that  method  also  exist  in  SAM  due  to  the  different  model 
representations  used  by  each  method.  This  document  is  a  process 
description  for  SAM,  not  a  training  manual.  Some  materials  are 
included  that  might  support  appraisal  training  or  the  development  of 
appraisal  training  materials;  however,  it  is  not  the  intent  of  this 
document  to  be  a  substitute  for  appraisal  training. 


How  should  it  be  The  SAM  description  is  written  to  support  facilitated  self-appraisal 
used?  against  the  SE-CMM  as  a  reference  model.  This  SAM  description  was 

not  written  to  meet  the  needs  of  a  third-party  evaluation,  although  it 
could  be  adapted  to  that  environment  by  experienced  appraisers  if 
necessary.  Third-party  use  of  the  method  is  not  encouraged  by  the  SE- 
CMM  Steering  Group  as  of  this  version's  release. 


continued  on  next  page 


'  CMM  and  Capability  Maturity  Model  are  registered  service  marks  of  Carnegie  Mellon  University. 
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To  th©  Roador,  Continued 


Who  developed 
SAM? 


Intended 

audience 


SE-CMM 
steering  group 
members 


SAM  was  developed  by  the  SE-CMM  collaboration  members,  including 
Hughes  Aircraft  Company,  Lockheed  Corporation,  Loral  Federal 
Systems  Company,  Loral  Space  &  Range  Systems,  Software 
Engineering  Institute,  Software  Productivity  Consortium,  and  Texas 
Instruments  Incorporated,  as  part  of  their  initial  effort  to  support 
community-wide  improvement  of  systems  engineering  processes. 


The  SE-CMM  is  focused  on  four  primary  groups:  systems  engineering 
practitioners  from  any  business  sector  or  government,  process  developers, 
individuals  charged  with  appraising  how  specific  systems  engineering 
organizations  implement  their  systems  engineering  processes,  and  systems 
engineering  managers.  Persons  with  five  years  or  more  of  experience  as  a 
systems  engineering  practitioner  or  manager  and  exposure  to  formal 
methods  of  organization  assessment  will  benefit  most  from  the  model  and 
appraisal  method. 


The  1994  Steering  Group  for  the  SE-CMM  Project  has  provided  both 
traditional  management  oversight  functions  and  extensive  technical  and 
strategic  input  to  the  project,  and  their  individual  and  collected 
contributions  to  the  project  are  appreciated  beyond  measure.  The  names 
and  organization  of  the  SE-CMM  Steering  Group  members,  as  of  May 
1995,  are  provided  in  the  table  below: 


Organization 

Contacts 

Department  of  Defense/OSD 

John  Burt 

Hughes  Aircraft  Company 

Dene  Minnich 

Lockheed  Martin  Corporation 

Michael  Carroll 

Loral  Federal  Systems  Company 

Gary  Kennedy 

National  Institute  of  Standards  and 

Technology 

Roger  Martin 

National  Council  on  Systems  Engineering 

Don  Crocker 

Software  Engineering  Institute 

William  Peterson 

Software  Productivity  Consortium 

Art  Pyster,  PhD 

Texas  Instruments,  Incorporated 

Merle  Whatley,  PhD 

European  Software  Institute 

Colin  Tully,  PhD 

SE-CMM  Collaboration  Contacts 
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To  th©  Roador,  Continued 


Additional 
information- 
project  office 


If  you  have  any  questions  about  this  method  or  about  pilot  appraisals  using 
the  SE-CMM,  please  contact  the  SE-CMM  Project.  The  maintenance  site 
for  the  project  is  the  Software  Engineering  Institute  of  Carnegie  Mellon 
University.  For  information,  contact  SEI  Customer  Relations  or  Hal 
Pierson,  the  project  manager  at: 


Customer  Relations 
Software  Engineering  Institute 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213 
(412)268-5800  (voice) 
(412)268-5758  (fax) 
customer-relations  @  sei.cmu.edu 


Hal  Pierson 

Software  Productivity  Consortium 
SPC  Building 
2214  Rock  Hill  Road 
Herndon,  VA  22070 
1-800-827-4772  (voice) 

(703)  742-7200  (fax) 
ask-spc  @  software.org 


Data  rights 
associated  with 
the  SE-CMM 


The  SE-CMM  collaboration  members  encourage  free  use  of  the  SE- 
CMM  Appraisal  Method  Description  as  a  reference  for  the  systems 
engineering  community.  Members  have  agreed  that  this  and  future 
versions  of  this  document,  when  released  to  the  public,  will  retain  the 
concept  of  free  access  via  a  permissive  copyright  notice. 
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IX 


Chapter  1 :  Introduction  to  the  SE-CMM  Appraisal 
Method 


Overview  of 
document 


In  this  chapter 


This  document  contains  basic  information  on  the  SE-CMM  a.ppraisal 

method  (SAM).  It  is  broken  into  three  chapters  with  appendices: 

•  Chapter  1  contains  basic  context  information  and  assumptions  used  in 
creating  the  method. 

•  Chapter  2  contains  descriptions  of  each  of  the  major  process  elements 
of  SAM. 

•  Chapter  3  contains  guidance  information  that  is  helpful  in  preparing 
for  and  conducting  an  appraisal. 

•  The  appendices  contain  templates  and  instructions  for  using  support 
tools  recommended  in  SAM,  as  well  as  the  SAM  questionnaires. 


The  following  table  provides  a  guide  to  the  information  found  in  this 
chapter. 


Topic 

See  Page 

1 . 1  Summary  of  the  SE-CMM  Appraisal  Method 

1-2 

1.2  Assumptions 

1-9 

1 .3  Appraisal  Roles 

1-10 

1 .4  Appraisal  Instruments 

1-14 
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1.1  Summary  of  the  SE-CMM  Appraisal  Method 


Introduction 


Appraisal 

initiation 


Typical  appraisal 
purposes 


Tailoring 
appraisal  based 
upon  objectives 


This  section  provides  a  brief  overview  of  important  aspects  of  initiating, 
preparing,  and  conducting  a  SAM.  Each  of  these  issues  is  treated  in 
more  depth  either  in  Chapter  2  or  Chapter  3. 


Appraisals  are  typically  performed  in  organizations  either  for  self 
improvement  or  supplier  selection.  Since  this  appraisal  method  is  based 
on  the  first  release  of  the  SE-CMM,  using  SAM  for  supplier  selection  is 
not  recommended. 

Process  improvement  appraisals  can  vary  significantly  depending  upon 
the  specific  objectives  of  the  appraisal.  Appraisal  objectives  will  have 
an  impact  primarily  on  the  selection  of  the  participants  and  the  duration 
of  the  appraisal  period.  Factors  to  consider  in  formulating  appraisal 
objectives  are  covered  in  Chapter  2,  Sections  2. 1.1 -2. 1.3. 


The  purpose  of  appraisal  is  typically  one  of  the  following: 

•  Identify  specific  areas  for  improvement  based  upon  known  general 
areas  of  deficiency. 

•  Confirm  known  data  on  systems  engineering  practices. 

•  Obtain  buy-in  for  change  from  the  organization. 

•  Confirm  process  improvement  progress  and  determine  new  status 
(typically  second  or  subsequent  appraisals). 


The  appraisal  should  be  tailored  to  meet  the  objectives  of  the 
organization.  Appraisals  can  be  focused  on  almost  any  cohesive 
business  unit  of  an  enterprise:  a  specific  project,  program,  strategic 
business  unit,  functional  organization,  multiple  functional  organizations, 
or  the  entire  company.  The  selection  is  based  upon  the  sponsoring 
organization's  goals.  The  appraisal  can  also  be  focused  on  addressing 
all  of  the  process  areas,  which  is  typical  for  an  initial  appraisal,  or  it  may 
focus  on  specific  process  areas  that  may  have  been  the  target  for  process 
improvement.  Tailoring  the  number  of  process  areas  and  the  scope  of 
the  entity  being  appraised  (e.g.,  project,  functional  organizations,  etc.) 
are  the  two  aspects  that  will  have  a  si^ificant  impact  on  the  duration 
and  resources  required  for  the  appraisal. 

Specific  aspects  of  the  appraisal  that  can  be  tailored  are  addressed  in  the 
"Tailorable  Parameters"  block  of  each  process  element  described  in 
Chapter  2. 


continued  on  next  page 
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1.1  Summary  of  the  SE-CMM  Appraisal  Method,  Continued 


Recording 

tailoring 

information 


Results 


In  tailoring  any  appraisal  method,  it  is  important  to  note  what  aspects 
and  parameters  of  the  method  have  been  tailored,  so  that  people  who  use 
the  data  gathered  from  that  appraisal  can  understand  the  context  from 
which  the  data  were  collected.  This  tailoring  information  should  be 
recorded  in  the  appraisal  plan. 


The  primary  work  products  of  an  SE-CMM  appraisal  are  a  findings 
briefing  and  an  appraisal  report.  The  findings  briefing  is  presented  at 
the  end  of  the  on-site  period  of  the  appraisal  and  will  include  a  process 
capability  profile  and  appraisal  findings,  which  address  both  strengths 
and  weaknesses  of  the  appraised  entity.  The  weaknesses  are  typically 
limited  to  approximately  seven  synthesized  findings  to  keep  the  scope  of 
issues  to  a  number  that  the  organization  can  manage.  The  appraisal 
report  is  written  by  the  appraisal  team  after  the  on-site  period;  it  includes 
more  detail  on  each  of  the  findings  and  specific  recommendations  for 
process  improvement  focused  on  the  findings. 

Chapter  3  discusses  the  process  capability  profile  and  other  aspects  of 
developing  appraisal  results. 


continued  on  next  page 
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1 .1  Summary  of  the  SE-CMM  Appraisal  Method,  continued 


Labor 

requirements 


Phases 


Table  1-1  defines  typical  labor  requirements  for  a  complete  appraisal 
(e.g.,  all  SE-CMM  process  areas  applied  to  three  to  four  projects  or 
equivalent).  Labor  requirements  can  be  tailored  as  a  function  of  the 
scope  of  the  appraisal.  Typical  schedule  and  labor  templates  are 
included  in  Appendix  D.  The  hours  for  the  facilitator  include  the  time 
for  all  phases.  The  hours  for  other  roles  are  for  the  on-site  phase  only. 
Descriptions  of  the  appraisal  roles  are  found  in  the  section  "Appraisal 
Roles"  found  later  in  this  chapter. 


Role 

Recommended 
number  of 
people 

Hours  per 
person 

Total  hours 
for  this  role 

Facilitator 

2 

60-76 

120-152 

Appraisal  team 
member  (in 
addition  to 
facilitators) 

4-6 

50-66 

200-396 

Systems 

engineering  leads 

1  per  project  (3 
projects) 

6-8 

18-24 

Practitioners 
from  across 
organization 

30  (3  groups  of 
10) 

6-8 

180-240 

TOTAL 

518-812 

Table  1-1.  Labor  Requirements  for  an  Appraisal. 


Table  1-2  lists  the  phases  of  the  appraisal  process.  The  process 
elements  for  each  phase  are  fully  described  in  Chapter  2. 


Phase 

Description 

Preparation 

The  activities  done  in  preparation  for  an  appraisal 

On-site 

The  activities  done  at  the  site  of  the  entity  being 
appraised 

Post  Appraisal 

The  activities  done  after  the  on-site  appraisal 
period 

Table  1-2.  Appraisal  Phases. 


continued  on  next  page 
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1.1  Summary  of  the  SE-CMM  Appraisal  Method,  Continued 


Diagram  of  Figure  1-1  summarizes  the  steps  in  the  Preparation  phase. 

Preparation 

Phase 


Figure  1-1.  Diagram  of  Preparation  Phase. 


continued  on  next  page 
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1 .1  Summary  of  the  SE-CMM  Appraisal  Method,  Continued 


Diagram  of  on¬ 
site  Phase 


Figure  1-2  summarizes  the  steps  in  the  On-Site  phase. 


Figure  1-2.  Diagram  of  On-Site  Phase. 


continued  on  next  page 
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1.1  Summary  of  the  SE-CMM  Appraisal  Method,  Continued 


Diagram  of  Post- 
Appraisal  Phase 


Figure  1-3  summarizes  the  steps  in  the  Post- Appraisal  Phase. 


Figure  1-3.  Diagram  of  Post-Appraisal  Phase. 


continued  on  next  page 
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1.1  Summary  of  the  SE-CMM  Appraisal  Method,  Continued 


Relationship 

CBA-IPI 


to  For  those  readers  familiar  with  the  SEI  CBA-IPI  method,  Table  1  -3 
shows  the  correlation  between  the  appraisal  steps  documented  in 
Chapter  2  and  the  CBA-IPI  method. 


SAM  Step 

IPI  Step 

1 . 1  Obtain  Sponsor  Commitment 

1.2  Select  Appraisal  Parameters 

1.3  Plan  Appraisal  Details 

Plan  Assessment 

1.3  Plan  Appraisal  Details 

2. 1  Conduct  Opening  Meeting 

2.2  Familiarize  Team  with  SAM 

Prepare  for  Assessment 

1.4  Collect  Questionnaire  Data 

1.5  Analyze  Data 

2.3  Analyze  Questionnaire 

2.4  Interview  Systems  Engineering  Leads 

2.6  Interview  Practitioners 

Gather  Data 

2.5  Consolidate  Data  from  SE  Leads 

2.7  Consolidate  Data  from  Practitioners 

2.8  Develop  Preliminary  Findings 

2.9  Review  Preliminary  Findings 

2.10  Develop  Draft  Rating 

2. 1 3  Adjust  Draft  Findings 

Consolidate  Data 

2.5  Consolidate  Data  from  SE  Leads 

2.7  Consolidate  Data  from  Practitioners 

2.10  Develop  Draft  Rating 

Review/Revise  Data 
Gathering  Plans 

2.10  Develop  Draft  Rating 

Make  Rating  Judgments 

2.12  Present  Draft  Findings 

2.14  Present  Final  Briefing 

2.15  Brief  Sponsor  (optional) 

2.16  Conduct  Wrap-Up 

3 . 1  Report  Lessons  Learned 

3.2  Develop  and  Present  Action  Plans 

3.3  Perform  Actions 

Report  results 

Table  1-3.  Relationship  of  SAM  &  IPI  Activities. 
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1.2  Assumptions 


Introduction 


Assumptions 


SE-CMM  as 
reference  model 


Facilitator 

training 


SAM  focuses  on 
self-improvement 


This  section  discusses  the  underlying  assumptions,  both  procedural  and 
model-based,  that  are  relevant  to  using  SAM. 


The  SAM  description  is  based  on  the  following  assumptions: 

•  The  SE-CMM  is  the  reference  model  for  the  SAM. 

•  Readers  are  familiar  with  the  content  and  concepts  of  SECMM-94- 
04ICMU/SEI-94-HB-04,  A  Systems  Engineering  Capability  Maturity 
Model,  Version  1.0. 

•  Facilitators  intending  to  use  the  SAM  have  been  trained  in  basic 
organizational  appraisal  techniques. 

•  Although  SAM  can  be  tailored  to  either  self-improvement  or  supplier 
selection,  the  focus  and  tone  of  the  material  is  biased  toward  self- 
improvement. 


Although  the  basic  appraisal  method  described  herein  shares  many 
features  with  other  organizational  appraisal  methods,  the  data  gathering 
instruments  and  rating  development  process  are  specifically  related  to 
the  SE-CMM.  The  SE-CMM  is  fully  described  in  A  Systems 
Engineering  Capability  Maturity  Model,  Version  i.O  [SECMM].  This 
document  does  not  contain  specific  information  on  the  SE-CMM. 


SAM  is  an  appraisal  method  that  makes  heavy  use  of  informal 
knowledge  that  practitioners  provide  through  a  series  of  interviews  and 
feedback  sessions.  There  are  many  methods  for  interviewing  and 
synthesizing  data  from  these  types  of  data  sources.  Although  it  is  not 
necessary  for  all  members  of  the  appraisal  team  to  be  thoroughly  versed 
in  the  concepts  of  organizational  appraisal,  the  appraisal  will  proceed 
more  smoothly,  and  is  likely  to  produce  better  results,  if  at  least  one, 
and  preferably  two,  of  the  appraisal  team  members  are  skilled  in  the 
facilitator  role  for  organizational  appraisals.  The  facilitators  are  included 
as  members  of  the  appraisal  team. 


The  actual  character  of  each  appraisal  will  differ  based  on  differences  in 
organizations’  cultures  and  other  business  contexts.  The  process 
described  focuses  on  a  feedback  loop  that  includes  multiple  levels  of  the 
entity  being  appraised.  This  is  one  of  the  ways  in  which  the  SAM 
contributes  to  an  overall  process  improvement  focus  in  the  organization. 
There  are  many  other  models  for  organizational  improvement,  and  most 
include  a  benchmarking  step  that  serves  to  determine  the  current  state  of 
the  organization.  Many,  like  SAM,  use  this  benchmarking  step  as  a 
way  to  focus  attention  and  resources  on  the  improvement  effort. 
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1.3  Appraisal  Roles 


Personnel  and  Table  1-4  defines  the  personnel  and  associated  roles  that  are  typically 
associated  roles  involved  in  an  appraisal.  The  responsibilities  and  functions  performed 
by  each  of  the  roles  is  further  defined  in  Chapter  2.  Note  that  multiple 
roles  may  be  assigned  to  one  person  as  appropriate. 


Role  Name 

Description 

Appraisal 

team 

The  appraisal  team  consists  of  those  who  conduct  the 
appraisal.  All  of  the  personnel  on  the  appraisal  team 
should  be  familiar  with  the  SE-CMM  prior  to  the  on¬ 
site  phase.  This  group  includes  the  following  roles: 

•  Facilitator. 

•  Appraisal  team  leader. 

•  Site  coordinator. 

•  Appraisers  drawn  from  the  entity  being  appraised 
(internal)  or  customer  (supplier  selection). 

The  term  appraisal  team  members  indicates  any  of  the 
roles  cited  above. 

Facilitator 

Member  of  the  appraisal  team  who  is  responsible  for 

•  Familiarizing  the  appraisal  team  with  SAM. 

•  Facilitating  the  appraisal  process  during  the  on-site 
phase. 

•  Providing  SE-CMM  expertise. 

The  facilitator  is  often  drawn  from  outside  the 
sponsoring  organization. 

Appraisal 
team  leader 

Individual  who  is  responsible  for 

•  Presenting  the  appraisal  findings  and  developing  the 
appraisal  report. 

Site 

coordinator 

Individual  who  is  responsible  for 

•  Obtaining  facilities  for  the  on-site  phase. 

•  Scheduling  activities  during  the  on-site  week. 

•  Administering  and  collecting  questionnaires. 

•  Ensuring  personnel  attendance,  as  appropriate. 

Appraiser 

An  appraisal  team  member  who  participates  in  the  data 
collection,  analysis,  findings  generation,  and 
communication  with  the  participants  from  the 
organization  (appraised  entity). 

Table  1-4.  Appraisal  Roles. 


continued  on  next  page 
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1.3  Appraisal  Roles,  continued 


Personnel  and 
associated  roles 
(continued) 


Role  Name 

Description 

Appraisal 

participants 

The  appraisal  participants  are  the  subjects  of  data 
gathering  in  an  appraisal.  This  group  includes 

•  Project  lead  systems  engineers  (or  equivalent). 

•  Practitioners  (both  technical  and  management). 

•  Support  personnel. 

Participants  are  the  primary  sources  of  data  via 
questionnaires  and  interviews.  Guidance  on  participant 
selection  is  found  in  Chapter  2. 

Project  lead 
systems 
engineers  (or 
equivalent) 

Appraisal  participants  identified  as  having 
responsibility  for  the  systems  engineering  aspects  of  a 
project.  The  SE  lead  should  have  broad  knowledge  of 
the  full  life  cycle  of  product  development.  The  systems 
engineering  leads 

•  Complete  the  SAM  questionnaire. 

•  Participate  in  a  series  of  question  and  answer 
sessions. 

•  Are  a  primary  source  of  feedback  on  the  validity  of 
the  findings. 

Practitioners 

Individuals  who  perform  or  support  the  systems 
engineering  process  (direct  and  indirect,  e.g.,  training, 
customers,  and  suppliers).  Practitioners  are  a  source 
of  data,  primarily  via  the  interview  and  also  as  a 
reviewer  of  the  appraisal  findings.  They  could  also 
include  questionnaire  respondents. 

Appraisal 

support 

personnel 

Personnel  who  support  the  appraisal  process. 

Examples  include  secretaries  and  logistics  coordinators 
for  the  appraisal. 

Appraisal 

customer 

Individual  or  group  who  defines  the  objectives, 
receives  the  results,  and  covers  the  cost  of  the 
appraisal.  This  group  includes 

*  Sponsor 

•  Management 

Table  1-4.  Appraisal  Roles,  continued. 
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1 .3  Appraisal  Roles,  continued 


Personnel  and 
associated  roles 
(continued) 


Role  Name 

Description 

Sponsor 

Individual  providing  the  resources  for  the  appraisal  and 
the  commitment  to  the  process  improvement  effort,  in 
the  case  of  self-improvement.  It  is  particularly 
important  for  the  sponsor  to  show  commitment  by 
attending  both  the  opening  and  closing  meetings  of  the 
on-site  phase. 

Management 

Management  of  the  appraised  entity,  both  at  the 
organization-  and  project-level.  They  are  recipients  of 
the  appraisal  findings  and  the  primary  role  responsible 
for  carrying  out  improvements  suggested  by  the 
appraisal. 

Appraisal 

champion 

Individual  who  initiates  the  dialogue  within  the  sponsor 
and  management  groups  and  obtains  the  initial 
sponsorship  for  the  appraisal.  The  champion  usually 
plays  a  role  in  interfacing  the  site  coordinator  and  the 
sponsor  in  removing  any  obstacles  to  the  appraisal  that 
may  be  encountered. 

Table  1-4.  Appraisal  Roles,  continued. 
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1.4  Appraisal  Instruments 


Findings  The  goal  of  developing  the  final  findings  is  to  accurately  represent  the 

organization's  opportunities  for  process  improvement.  In  order  to 
accomplish  this,  data  is  sampled  from  the  organization  using  three 
complimentary  methods;  questionnaire,  interviews,  and  feedback 
sessions. 

The  findings  evolve  throughout  the  week  beginning  with  the  large 
amount  of  data  from  the  questionnaire.  From  this  data  and  data  from  the 
interviews,  a  set  of  preliminary  findings  is  distilled.  After  reviewing 
them  with  the  SE  Leads  the  preliminary  findings  are  further  distilled  to  a 
smaller  set  of  draft  findings.  Finally,  ^ter  reviewing  these  draft 
findings  with  both  the  SE  Leads  and  the  Practitioners,  the  appraisal  team 
generates  the  final  findings.  See  Figure  1-4,  Findings  Flow  below. 


Imerview  Practitioners 
22.6 


Interview  SE  Leads 
2-2.4 


Present  Draft  Findings 
Jo  Practitioners  -  2.2.12 


Review  Preliminary  Findings  to  SE  Leads  -  2.2.12 


2.2.9 


Figure  1-4.  Findings  Flow. 
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Findings 

Development 


As  the  findings  evolve  throughout  the  appraisal,  four  identifiable 
products  are  created.  These  are  described  in  the  following  table.  (See 
Appendix  J) 


Artifact 

Role  in  Appraisal  Process 

Questionnaires 

The  questionnaires,  representing  data  from  3 
or  4  projects,  serve  as  a  starting  point  for  the 
data  gathering  process.  The  questionnaire 
(see  Appendix  J)  provides  a  direct  link  to  the 
SE-CMM  model.  This  large  volume  of  data 
is  entered  into  the  Data  Tracking  Sheet,  (see 
Appendix  C)  Analysis  of  the  questionnaire 
leads  to  the  exploratory  questions  for  the  SE 
Leads. 

Preliminary  Findings 

The  preliminary  findings  are  the  appraisal 
team's  first  attempt  at  reducing  the 
information  that  it  has  gathered  (i.e.,  from  the 
questionnaire,  SEI  Lead  interviews  and 
Practitioner  Group  discussions)  into  a  Lead 
interviews  and  Practitioner  Group 
discussions)  into  a  set  of  observations. 
Typically  these  findings  are  not  well  worded, 
nor  do  they  represent  a  consensus  of  the 
team. 

Draft  Findings 

The  draft  findings  are  developed  based  on 
feedback  that  the  SE  Leads  supply  based  on 
their  review  of  the  preliminary  findings.  The 
draft  findings  are  the  apprais^  team's 
consensus  description  of  the  opportunities  the 
organization  has  for  process  improvement. 
They  are  carefully  worded,  and  include  the 
finding,  an  identifiable  cause,  and  a  business 
consequence. 

Final  Findings 

The  final  findings  are  developed  based  on  the 
feedback  that  the  Practitioners  and  the  SE 
Leads  supply  based  on  their  independent 
review  of  the  draft  findings.  The  typical 
kinds  of  changes  between  the  draft  and  final 
findings  are  wording  changes  for  community 
acceptance  or  clarifications  of  causes  or 
consequences. 
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^erview  Practitioners 

2.2.6 


\ 


A 


Questionnaire 
Rating  Algorithm 


2.3.3 


/ 


Interview  SE  Leads 
2.2.4 


Preliminary 

Rating 


Review  Preliminary  Findings 
2.2.9 


Present  Draft  Findings 
ITo  Practitioners  -  2,2,12 


Present  Draft  Findings 
to  SE  Leads  -  2.2.12 


Figure  1-5.  Ratings  Flow. 
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Ratings 

Development 


As  the  ratings  evolve  throughout  the  appraisal,  four  separate  estimates 
of  the  organizational  ratings  are  created.  These  estimates,  and  their 
development  throughout  the  week,  are  described  in  the  following  table. 
The  thrust  of  this  process  is  to  converge  on  a  set  of  final  ratings  that 
fairly  represent  the  team  consensus  of  the  organization's  capability 
profile. 


Estimate 

Role  in  Appraisal  Process 

Questionnaire  Rating 

The  questionnaire  data  from  the  3  or  4 
projects  is  entered  into  the  Data  Tracking 

Sheet,  from  which  the  initial  estimated  rating 
can  be  calculated. 

Exploratory  questions  are  developed  to 
resolve  discrepancies  or  inconsistencies  in  the 
data  from  the  questionnaires. 

Preliminary  Rating 

The  second  estimate  is  achieved  upon 
incorporating  answers  from  exploratory 
questions  and  data  from  the  Practitioner 
interviews. 

Each  incorporated  answer  is  intended  to 
address  a  discrepancy  or  inconsistency 
among  the  original  data  in  the  questionnaire. 
Consequently,  the  subsequent  columns  in  the 
Data  Tracking  Sheet  are  annotated  to  indicate 
whether  the  aggregated  data  indicates 
compliance  or  non-compliance  with  SE- 
CMM. 

Draft  Rating 

The  third  estimate  is  achieved  upon 
incorporating  feedback  from  the  SE  Leads 
based  on  their  review  of  the  Preliminary 
Findings. 

Final  Rating 

The  final  estimate  typically  is  identical  to  the 
draft  rating. 
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Ratings 

Development 


As  the  ratings  evolve  throughout  the  appraisal,  four  separate  estimates 
of  the  organizational  ratings  are  created.  These  estimates,  and  their 
development  throughout  the  week,  are  described  in  the  following  table. 
The  thrust  of  this  process  is  to  converge  on  a  set  of  final  ratings  that 
fairly  represent  the  team  consensus  of  the  organization's  capability 
profile. 


Estimate 

Role  in  Appraisal  Process 

Questionnaire  Rating 

The  questionnaire  data  from  the  3  or  4 
projects  is  entered  into  the  Data  Tracking 

Sheet,  from  which  the  initial  estimated  rating 
can  be  calculated. 

Exploratory  questions  are  developed  to 
resolve  discrepancies  or  inconsistencies  in  the 
data  from  the  questionnaires. 

Preliminary  Rating 

The  second  estimate  is  achieved  upon 
incorporating  answers  from  exploratory 
questions  and  data  from  the  Practitioner 
interviews. 

Each  incorporated  answer  is  intended  to 
address  a  discrepancy  or  inconsistency 
among  the  original  data  in  the  questionnaire. 
Consequently,  the  subsequent  columns  in  the 
Data  Tracking  Sheet  are  annotated  to  indicate 
whether  the  aggregated  data  indicates 
compliance  or  non-compliance  with  SE- 
CMM. 

Draft  Rating 

The  third  estimate  is  achieved  upon 
incorporating  feedback  from  the  SE  Leads 
based  on  their  review  of  the  Preliminary 
Findings. 

Final  Rating 

The  final  estimate  typically  is  identical  to  the 
draft  rating. 
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Chapter  2:  Summaries  of  SAM  Process  Elements 


Introduction 


In  this  section 


This  chapter  contains  summaries  of  the  major  phases  and  process 
elements  of  the  SAM. 


The  following  process  elements  are  addressed  in  this  chapter: 


ID 

Process  Element  Name 

See  Page 

2.1 

Preparation 

2-3 

2.1.1 

Obtain  Sponsor  Commitment 

2-6 

2.1.2 

Select  Appraisal  Parameters 

2-8 

2.1.3 

Plan  Appraisal  Details 

2-10 

2.1.4 

Collect  Questionnaire  Data 

2-12 

2.2 

On  Site 

2-14 

2.2.1 

Conduct  Opening  Meeting 

2-20 

2.2.2 
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Chapter  2: 

Continued 


How  to  read  the 
process  element 
summaries 


Summaries  of  SAM  Process  Elements, 


All  process  element  descriptions  are  similarly  formatted,  and  contain 
several  blocks,  which  are  described  in  Table  2-1  below.  For  each  major 
phase  (preparation,  on-site,  post-appraisal),  the  summary  description 
contains  the  process  elements  that  comprise  that  phase;  for  the  other 
process  elements,  the  process  element  summary  contains  the  actual 
description  of  that  element. 


Block  Title 

Description 

Element  title/tag 

The  SAM  title  for  the  process  element,  the  code 
for  the  process  element,  and  the  number  of  the 
element  used  for  reference 

Purpose 

The  major  purpose  for  this  process  element  within 
SAM 

Major  activities 

An  overall  summary  of  the  activities  associated 
with  the  process  element 

Major  participants 

The  major  roles  involved  in  the  process  element 
and  a  summary  of  their  responsibilities  associated 
with  the  process  element 

Typical  duration 

A  range  of  the  typical  time  duration  (e.g.,  number 
of  hours)  expected  for  the  process  element 
described 

Steps 

The  substeps  for  the  element  along  with  guidance 
for  their  performance,  if  appropriate 

Tailorable 

parameters 

The  parameters  associated  with  this  process 
element  that  are  expected  to  be  tailored  for 
different  appraisal  goals 

Exit  criteria 

A  description  of  the  decision-making  criteria  to 
determine  if  the  process  element  has  been 
completed 

Notes 

Notes  on  the  process  element  that  do  not  fit  in  any 
of  the  other  categories 

Table  2-1.  How  to  Read  Process  Element  Descriptions. 
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2.1  Preparation 


Element 

tag 

Purpose 


Diagram 


title/  2. 1  Preparation  (PR) 


The  purpose  of  the  Preparation  phase  is  to  prepare  the  sponsor,  the 
appraisal  team,  and  the  appraised  entity  for  the  on-site  period.  Major 
elements  of  preparation  include 

•  The  sponsor  must  commit  to  the  resource  requirements  and  action 
requirements  for  the  appraisal  process. 

•  The  team  must  be  selected  to  perform  the  appraisal. 

•  The  appraised  entity  must  provide  preliminary  data  to  the  team  for 
analysis. 

•  The  te2im  must  analyze  the  data  prior  to  arriving  on-site. 

•  Figure  2-1  shows  the  major  steps  in  the  preparation  phase. 


Figure  2-1.  Diagram  of  Preparation  Phase. 
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2.1  Preparation,  Continued 


Summary 

description 


Table  2-2  lists  the  major  activities  of  the  Preparation  phase  and  the 
expected  output  of  each.  Each  element  is  described  more  fully  in  the 
summaries  that  follow. 


ID 

Activity 

Description 

Output 

2.1.1 

Obtain 

sponsor 

commitment 

Get  the  sponsor  to 

•  agree  to  the  concept 
of  the  appraisal 

•  define  goals  of  the 
appraisal  in  relation  to 
the  sponsor's 
business  goals 

•  agree  to  provide 
necessary  resources, 
including  funding 

•  understand  his/her 
role  in  sponsoring  the 
improvement  activity 
implied  by  engaging 
in  the  appraisd 

•  Appraisal 
resources 

•  Agreement  to 
proceed  and 
expend  resources 
for  the  planning 
and  conduct  of 
the  appraisal 

•  Appraisal  goals 
related  to 
business  goals 

2.1.2 

Select 

appraisal 

parameters 

Select  appraised  entity 
and  appraisal  team,  and 
tailor  appraisal  method 
as  required  (e.g.,  team 
size,  site  time,  ratings 
to  produce,  number  of 
projects,  project 
characteristics) 

•  Preliminary 
appraisal  plan  that 
defines  appraisal 
parameters  based 
on  appraisal  goals 

2.1.3 

Plan  appraisal 
details 

Use  appraised 
parameters  to  develop 
schedule,  training 
requirements,  and 
appraisal  support 
materials;  to  select 
training  instruments;  to 
develop  and  train 
appraisal  team 

•  Approved 
appraisal  plan 

2.1.4 

Collect  data 

Administer 
questionnaire  and 
review  any  appropriate 
documents/artifacts 

•  Responses  to 
questionnaire 

•  Document/artifact 
review  notes 

Table  2-2.  Summary  Description  of  Preparation  Phase. 
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2.1  Preparation,  continued 


Major 

participants 


Typical  duration 


Tailorable 

parameters 


Exit  criteria 


Notes 


Table  2-3  lists  the  primary  roles  involved  in  the  preparation  phase  and  a 
summary  of  their  activity  during  this  phase. 


Role 

Summary 

Sponsor 

Makes  the  decision  to  commit  to  appraisal  and 
provides  resource  commitments  and  tailoring 
guidelines  for  the  appraisal. 

Facilitator/site 

coordinator 

Interacts  with  the  management  and  personnel 
at  the  site  to  obtain  commitment  for  the 
appraisal  and  ensure  that  the  planning  for  the 
appraisal  is  successfully  completed. 

Appraisal  team 

Interacts  with  management  and  the  appraisal 
leader  to  perform  the  planning  and  other 
preparation  activities  for  the  appraisal. 

Table  2*3.  Participants  for  Preparation. 


Four  to  eight  weeks,  depending  on  the  complexity  of  the  appraisal 
selected  and  other  site  parameters 


•  Number  of  site  visits  prior  to  on-site  period. 

•  Number  of  meetings  with  senior  management. 

•  Number  of  on-site  visits. 

•  Duration  of  on-site  visit(s). 

•  Number  of  meetings  with  appraisal  team. 

•  Appraisal  goals. 

•  Number  of  projects  to  appraise. 

•  Number  of  appraisal  team  members. 

•  Process  area/capability  level  focus  for  the  appraisal. 


•  Appraisal  goals  established. 

•  Appraisal  resources  committed. 

•  Project,  team,  participant  selection  completed. 

•  Appraisal  plan  approved. 

•  Preliminary  data  gathering  completed. 


This  is  where  the  overall  context  for  the  appraisal  is  established.  The 
decisions  made  here  will  have  an  impact  on  the  rest  of  the  appraisal 
activities.  All  relevant  decisions  from  this  phase  should  be  documented 
in  the  appraisal  plan. 
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2.1.1  Obtain  Sponsor  Commitment 


Element  title/ 
tag 


Purpose 


Summary 

description 


Major 

participants 


2.1.1  Obtain  Sponsor  Commitment  (OC). 


The  purposes  of  Obtain  Sponsor  Commitment  are  to  establish  the 
sponsor's  commitment  to  the  appraisal  and  to  determine  the  major  goals 
of  the  appraisal,  which  will  guide  the  selection  of  appraisal  parameters. 


Obtain  Sponsor  Commitment  involves  meeting  with  the  sponsors  of  the 
appraisal  to  provide  an  understanding  of  the  concepts  of  the  SE-CMM 
and  the  SE-CMM  appraisal  method  (SAM),  engaging  the  sponsor  in 
dialogue  to  determine  his/her  goals  for  the  appraisal,  and  negotiating  a 
commitment  for  resources,  including  personnel  and  funding  for  the 
appraisal  activities.  A  primary  aspect  of  obtaining  sponsor  commitment 
is  ensuring  that  the  sponsor  understands  his/her  role  in  the  appraisal. 
For  tailoring  guidance,  please  read  the  related  chapter  3  paragraphs. 


Table  2-4  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Sponsor 

Engages  in  dialogue  with  appraisal 
leader/champion  to  understand  appraisal 
context  and  set  appraisal  goals;  commits 
resources  for  the  appraisal. 

Appraisal  champion 

Establishes  link  between  sponsor,  appraisal 
team  leader,  and  potential  appraisal 
participants. 

Appraisal  team 
leader 

Provides  sponsor  with  information  needed  to 
make  a  commitment  decision  on  the  appraisal; 
provides  a  link  between  the  appraisal 
champion  and  potential  appraisal  participants. 

Table  2-4.  Participants  for  Obtain  Sponsor  Commitment. 
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2.1.1  Obtain  Sponsor  Commitment,  Continued 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 

Notes 


Two  hours  to  several  weeks 


•  Method  of  interaction  with  sponsor:  all  face-to-face  meetings, 
combination  of  telephone/video  and  face  to  face  meetings, 
combination  of  written  communication  and  face-to-face  meetings,  etc. 

•  Number  of  interactions  with  sponsor. 


•  Appraisal  goals  established. 

•  Sponsor  commitment  to  provide  appraisal  resources  obtained. 

•  Sponsor  commitment  to  appropriate  behavior  during  the  appraisal 
obtained. 


This  is  a  go/no-go  decision  point.  If  sponsor  commitment  for  the 
appraisal  is  not  obtained,  no  further  process  elements  related  to  SAM 
will  be  performed. 

Guidance  on  using  business  goals  and  organization  context  for  tailoring 
the  appraisal  is  addressed  in  Chapter  3. 
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2.1.2  Select  Appraisal  Parameters 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 

Tailorable 

parameters 


2.1.2  Select  Appraisal  Parameters  (SP). 


The  purpose  of  Select  Appraisal  Parameters  is  to  determine  how  the 
SAM  needs  to  be  tailored  in  order  to  meet  the  goals  established  for  the 
appraisal  with  the  sponsor. 


Select  Appraisal  Parameters  involves  determining  the  membership  of  the 
appraisal  team,  the  profile  of  projects  to  be  selected  from  the  appraised 
entity,  and  the  major  appraisal  participants.  In  addition,  a  preliminary 
appraisal  plan  is  produced  that  documents  the  tailoring  of  SAM  process 
elements.  For  tailoring  guidance,  please  read  the  related  Chapter  3 
paragraphs. 


Table  2-5  lists  the  primary  roles  involved  in  this  process  element  and  the 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team  leader 
(ATL)  or  facilitator 

Engages  in  dialogue  with  the  appraised 
entity's  management  to  determine  the 
project  profile  for  the  appraisal;  engages 
with  the  appraising  organization  to 
determine  the  makeup  of  the  appraisal  team 
(in  the  case  of  self-improvement,  this  is  the 
same  as  the  appraised  entity).  Produces 
plan. 

Sponsor 

Engages  in  dialogue  with  ATL  and 
provides  appropriate  input  on  team  and 
appraised  entity  characteristics. 

Table  2-5.  Participants  for  Select  Appraisal  Parameters. 


One  to  two  weeks,  depending  on  complexity  of  tailoring  required 


•  Depth  of  documentation  of  tailoring  decisions. 
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2.1.2  Select  Appraisal  Parameters,  Continued 


Exit  criteria 


Notes 


•  Appraisal  team  leader  selected. 

•  Appraisal  team  selected. 

•  Projects  to  be  appraised  selected. 

•  Appraisal  participants  selected. 

•  Preliminary  appraisal  plan  developed. 


This  is  the  step  where  tailoring  decisions  are  made  and  documented 
based  on  the  goals  of  the  appraisal. 

Guidance  on  tailoring  the  appraisal  to  support  the  organization's 
business  goals  and  organizational  context  is  addressed  in  Chapter  3. 
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2.1.3  Plan  Appraisal  Details 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 

Tailorable 

parameters 


2.1.3  Plan  Appraisal  Details  (PD). 


The  purpose  of  Plan  Appraisal  Details  is  to  produce  and  obtain  approval 
for  the  final  appraisal  plan,  which  documents  the  parameters  and  details 
of  the  appraisal.  For  tailoring  guidance,  please  read  the  related  Chapter 
3  paragraphs. 


Plan  Appraisal  Details  involves  establishing  the  availability  of  planned 
interviewees  during  the  on-site  period,  planning  the  logistics  of  the 
appraisal  (meeting  rooms,  support  staff  availability,  etc.),  and  verifying 
the  schedule  for  the  appraisal  with  all  affected  parties.  It  also  involves 
verifying  who  will  receive  data  on  the  conclusions  of  the  appraisal  and 
establishing  feedback  mechanisms  for  lessons  learned.  For  tailoring 
guidance,  please  read  the  related  Chapter  3  paragraphs. 


Table  2-6  lists  the  primary  roles  involved  in  this  process  element  and  the 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team  leader 

Oversees  production  of  and  approval  for 
final  appraisal  plan. 

Appraisal  team 

Produces  assigned  sections  of  appraisal 
plan. 

Sponsor 

Approves  final  appraisal  plan. 

Appraisal  champion 

Stays  in  touch  with  site  coordinator  on 
progress  of  preparation  and  assists  in 
removing  obstacles  for  the  appraisal. 

Site  coordinator 

Verifies  the  schedules  of  the  intended 
participants,  arranges  logistics  details  for 
the  appraisal. 

Table  2-6.  Participants  for  Plan  Appraisal  Details. 


Two  to  six  weeks,  depending  on  the  complexity  of  the  appraisal  plan 


•  Degree  to  which  details  are  planned  (depends  on  the  complexity  of  the 
appraisal). 


continued  on  next  page 
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2.1.3  Plan  Appraisal  Details,  continued 


Exit  criteria 


Notes 


•  Appraisal  schedule  established. 

•  Appraisal  participant  availability  confirmed. 

•  Appraisal  plan  approved. 

•  Logistics  of  appraisal  prepared. 


This  is  when  the  actual  schedule  for  the  on-site  phase  is  produced.  (See 
Appendix  F,  Site  Coordinator  Checklist,  for  details  on  preparing  for  the 
logistics  of  an  appraisal.) 

If  the  appraisal  champion  has  not  become  an  appraisal  team  member, 
he/she  is  likely  to  keep  in  touch  with  the  appraisal  team  leader 
throughout  the  appraisal  preparation  phase. 

Any  useful  documents,  such  as  policies,  process  descriptions  (often  in 
training  materials),  standards,  even  meeting  minutes  or  action  item  lists, 
may  be  requested,  collected,  and  stored  for  use  by  the  appraisal  team. 
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2.1.4  Collect  Questionnaire  Data 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


2.1.4  Collect  Questionnaire  Data  (CQ). 


The  purpose  of  Collect  Questionnaire  Data  is  to  obtain  profile 
information  on  the  appraised  entity  and  to  administer  and  collect  data 
from  the  questionnaire.  Preliminary  transference  of  the  questionnaire  to 
the  DTS  can  also  be  performed  as  part  of  this  step  if  the  necessary 
resources  are  available. 


Collect  Questionnaire  Data  involves  administering  and  collecting  results 
from  the  questionnaire  and  instmments  used  to  profile  the  organization. 
The  questionnaire  rephrases  the  base  and  generic  practices  of  the  SE- 
CMM  into  a  form  that  is  appropriate  for  data  gathering.  Its  results  are 
used  to  focus  the  on-site  data  gathering  for  the  appraisal.  The 
questionnaire  responses  are  transferred  to  the  data  tracking  sheet  either 
in  this  step  or  in  the  "Analyze  Questionnaire"  step  in  Chapter  2.  For 
tailoring  guidance,  please  read  the  related  Chapter  3  paragraphs. 


Table  2-7  lists  the  primary  roles  involved  in  this  process  element  and  the 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  participants 

The  selected  participants  provide  data  on 
projects  and  the  organization  via  profile 
questions  and  the  questionnaire.  The 
questionnaire  may  be  administered  in 
various  ways,  e.g.,  all  participants 
together  in  a  face-to-face  meeting,  as  an 
on-line  response  activity,  or  mail-ins.  It  is 
recommended  that  the  questionnaire  be 
administered  to  the  group,  with  the 
facilitator  or  other  person  with  knowledge 
about  the  SE-CMM  present  to  answer 
questions  of  terminology,  etc. 

Site  Coordinator 

Determines  respondents  of  the 
questionnaires  and  may  oversee 
administration . 

Appraisal  team  leader  of 
facilitator 

Conducts  any  meetings  related  to 
introducing  the  data  collection  activity. 

Sponsor 

Ensures  that  all  participants  are  made 
available  for  data  collection. 

Table  2-7.  Participants  for  Collect  Questionnaire  Data. 
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2.1.4  Collect  Questionnaire  Data,  Continued 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 

Notes 


One  to  two  hours  per  participant 


•  Method  of  administering  questionnaires. 

•  Contents  of  questionnaire,  based  on  appraisal  goals  (e.g.,  if  only 
appraising  process  areas  1, 3, 7,  only  use  those  questionnaire  items). 

•  Amount  of  analysis  performed  prior  to  on-site  phase. 

•  Transference  of  questionnaire  responses  to  the  DTS. 


•  Project  profile  information  received  by  appraisal  team. 

•  Filled-in  questionnaires  received  by  appraisal  team. 

•  Optional:  if  it  is  decided  to  transfer  questionnaire  responses  prior  to 
the  on-site  phase,  the  transference  should  be  completed  at  the  end  of 
this  process  element. 


See  the  questionnaire  distribution  table  in  Section  3.3  and  the  SAM 
questionnaire  instructions  in  Appendix  F  and  J  for  information  on  using 
the  questionnaire  to  collect  data.  See  Section  2.2.3  for  details  on 
transferring  the  questionnaire  responses  to  the  DTS. 
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2.2  On  Site 


Element  title/tag 
Purpose 


2.2  On  Site 


The  purpose  of  the  On-Site  phase  is  to  explore  the  results  of  the 
preliminary  data  analysis,  provide  an  opportunity  for  practitioners  at  the 
appraised  entity  to  participate  in  the  data  gathering  and  validation 
process,  and  provide  practitioners  and  management  with  input  on  the 
results  of  the  appraisal.  Figure  2-2  shows  the  steps  in  the  On-Site 
phase. 

It  should  be  noted  that  the  On-Site  phase  may  occur  over  several  visits. 
Appendix  D  demonstrates  the  various  on-site  schedules  that  may  be 
followed. 


continued  on  next  page 
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2.2  On  Site,  Continued 


Diagram 


Figure  2-2.  Diagram  of  On-Site  Phase. 
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2.2  On  Site,  Continued 


Summary 

description 


Table  2-8  lists  the  major  activities  of  the  on-site  phase  and  the  expected 
output  of  each.  Each  element  is  described  more  fully  in  the  summaries 
that  follow. 


ID 

Activity 

Description 

Outputs 

2.2.1 

Conduct 

opening 

meeting 

The  opening  meeting 
provides  participants  with 
an  overview  of  the 
appraisal  and  reminds 
them  of  the  context  of  the 
appraisal  (e.g.,  schedule 
and  confidentiality).  It  is 
also  the  time  where  the 
sponsor  expresses  support 
of  the  appraisal  and 
subsequent  process 
improvement  activities. 

•  Supported 
appraisal  effort 

•  Answered 
questions 

2.2.2 

Familiarize 
team  with 
SAM 

The  facilitator  instructs  the 
appraisal  team  on  the 
detailed  conduct  of  the 
appraisal  activities  and 
introduces  the  use  of  the 
model  for  appraisal. 

•  Prepared  team 

2.2.3 

Analyze 

question¬ 

naire 

The  appraisal  team 
analyzes  the  responses  to 
the  questionnaire  and 
formulates  a  set  of  follow- 
on,  exploratory  questions 
for  the  systems 
engineering  leads. 

•  Exploratory 
questions 

2.2.4 

Interview 

systems 

engineering 

leads 

Through  structured 
interview  techniques  using 
the  exploratory  questions, 
the  appraisal  team  gathers 
corroborating  data 
regarding  the  project’s 
systems  engineering 
practices. 

•  Interview  notes 

•  Data  requests 

2.2.5 

Consolidate 
data  from  SE 
leads' 
interviews 

The  team  members  review 
their  notes,  discuss  any 
issues,  and  update  their 
data  tracking  sheets 
(DTSs). 

*  Updated  DTS 

•  Adjustments  to 
practitioner 
interviews 

Table  2-8.  Summary  Description  of  the  On-Site  Phase. 
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2.2  On  Site,  Continued 


Summary 

description, 

continued 


ID 

Activity 

Description 

Outputs 

2.2.6 

Interview 

practitioners 

Through  open-ended, 
facilitated  discussion,  the 
team  gathers  corroborating 
data  on  organizational 
practices  from  different 
types  of  practitioners. 

•  Interview  notes 

2.2.7 

Consolidate 
data  from 
practitioner 
interviews 

The  team  members  review 
their  notes,  discuss  any 
issues,  and  update  their 
data  tracking  sheets. 

•  Updated  DTS 

2.2.8 

Develop 

preliminary 

findings 

Using  all  data  sources 
available,  the  team 
generates  a  preliminary 
list  of  findings  with  regard 
to  the  organization's 
systems  engineering 
practices. 

•  Preliminary 
findings 

2.2.9 

Review 

preliminary 

findings 

The  team  provides 
preliminary  findings  to 
systems  engineering  leads 
to  validate  that  what  they 
heard  is  correct. 

•  Notes  from 
presentations 

•  Validated/cor¬ 
rected  findings 

2.2.1C 

Develop  draft 
rating 

The  team  members  review 
their  notes,  discuss  any 
issues,  update  their  data 
tracking  sheets,  and 
formulate  draft  ratings. 

•  Updated  DTS 

•  Draft  ratings 

2.2.11 

Develop  draft 
findings 

The  team  prioritizes  and 
provides  wording  for 
findings  that  fit  the 
appraisal  context. 

•  Prioritized  draft 
findings 

Table  2-8.  Summary  Description  of  the  On-Site  Phase,  continued 


continued  on  next  page 
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2.2  On  Site,  Continued 


Summary 

description, 

continued 


ID 

Activity 

Description 

Outputs 

2.2.12 

Present  draft 
findings 

The  team  reports  appraisal 
findings  to  the  appraisal 
practitioners  and  systems 
engineering  leads. 

•  Practitioner 
feedback 

•  Feedback  from 
systems 

engineering  leads 

•  Proposed 
adjustments  to 
findings 

2.2.13 

Adjust  draft 
findings 

Based  on  participants' 
feedback,  the  findings  are 
adjusted  for  final 
presentation.  At  this  time, 
the  process  area  (PA) 
ratings  are  reviewed  for 
appraisal  team  consensus. 

•  Final  briefing 

•  Findings 

•  PA  ratings 

2.2.14 

Present  final 
briefing 

The  final  findings  and 
process  capability  profile 
are  presented  to  all 
participants  in  the  presence 
of  the  sponsor.  Future 
activities  are  discussed. 

•  Participant  buy-in 

•  Participant 
expectations 

2.2.15 

Brief 

sponsor 

(optional) 

The  team  provides  the 
sponsor  with  the 
opportunity  to  discuss  the 
results  with  the  appraisal 
team  privately. 

•  Sponsor  buy-in 

•  Sponsor 
expectations 

2.2.16 

Conduct 

wrap-up 

The  appraisal  team 
discusses  post  on-site 
activities.  They  also 
provide  feedback 
regarding  the  SE-CMM 
and  SE-CMM  appraisal 
method  for  use  by  the  SE- 
CMM  collaboration. 

•  Follow-on  plans 

•  SE-CMM 
feedback 

Table  2-8.  Summary  Description  of  the  On-Site  Phase,  continued 


continued  on  next  page 
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2.2  On  Site,  Continued 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 


See  Appendix  D,  Sample  Schedules  for  the  On-Site 


•  Number  of  interviews. 

•  Number  of  visits. 

•  Number  and  type  of  document  reviews. 

•  Amount  of  direction  in  interviews. 


•  Interviews  completed. 

•  Document  reviews  completed. 

•  Findings  briefing  delivered. 

•  Appraisal  report  planned. 

•  Feedback  on  SE-CMM  and  SAM  collected. 
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2.2.1  Conduct  Opening  Meeting 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2. 1  Conduct  Opening  Meeting  (OM) 


The  purpose  of  Conduct  Opening  Meeting  is  to  present  the  appraisal 
process  and  schedule  to  the  sponsor  and  dl  appraisal  participants.  An 
additional  purpose  is  for  the  sponsor  to  express  support  for  the  appraisal 
activities. 


Conduct  Opening  Meeting  involves  gathering  all  the  appraisal 
participants  together,  along  with  the  sponsor  or  customer  of  the 
appraisal  (depending  on  appraisal  context),  to  review  the  appraisal 
process  and  reaffirm  the  sponsor’s  commitment  to  the  appraisal. 


Table  2-9  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team  leader 

Welcomes  participants  and  introduces 
sponsor. 

Sponsor 

Shows  management  support  for  the 
appraisal  and  subsequent  process 
improvement  activities. 

Facilitator 

Presents  brief  overview  of  the  model  and 
appraisal. 

Site  coordinator 

Presents  schedules  and  locations. 

Appraisal  team 

Supports  appraisal  team  leader. 

Appraisal  participants 

Learn  their  role  in  the  on-site  phase  of 
appraisal  activities. 

Table  2-9.  Participants  for  Conduct  Opening  Meeting. 


1.5  hours 


continued  on  next  page 
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2.2.1  Conduct  Opening  Meeting,  Continued 


Steps 


Tailorable 

parameters 


Exit  criteria 


Notes 


Table  2-10  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Sponsor  comments 

The  sponsor  expresses  support  for  the 
appraisal  and  commitment  to  the  resulting 
process  improvement  recommendations. 

SE-CMM  introduction 

The  facilitator  gives  a  brief  introduction  to 
the  SE-CMM. 

Appraisal  process 

The  facilitator  presents  an  overview  of  the 
appraisal  process.  Confidentiality  rules  are 
explained. 

Schedule  review 

The  site  coordinator  reviews  the  schedule 
and  locations  for  the  week’s  activities  and 
stresses  the  necessity  of  being  on  time. 

Question  &  answer 

The  sponsor,  facilitator,  and  appraisal  team 
leader  answer  any  questions  from  the 
appraisal  participants. 

Table  2-10.  Steps  for  Conduct  Opening  Meeting. 


•  The  presentation  will  vary  based  on  the  purpose  of  the  appraisal.  In 
any  case,  the  way  in  which  the  results  will  be  used  should  be  a  part  of 
this  presentation. 


•  Opening  briefing  delivered. 

•  Questions  of  appraisal  participants  answered. 


Refer  to  Appendix  A  for  a  sample  opening  briefing. 

The  appraisal  goals  and  use  of  results  is  established  in  the  preparation 
phase  as  part  of  Select  Appraisal  Parameters. 
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2.2.2  Familiarize  Team  with  SAM 


Element  title/tag 
Purposes 

Summary 

description 

Major 

participants 


Typical  duration 


2.2.2  Familiarize  Team  with  SAM  (FT) 


The  purpose  of  Familiarize  Team  with  SAM  is  to  prepare  the  appraisal 
team  for  performing  its  role  in  the  appraisal  process. 


This  is  an  opportunity  for  the  appraisal  team  to  begin  to  work  together. 
The  briefing  begins  with  a  review  of  the  SE-CMM.  The  appraisal  steps 
are  presented  in  greater  detail  than  at  the  opening  meeting,  and  the 
team's  role  in  each  step  is  clarified. 


Table  2-1 1  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Presents  the  familiarization  briefing  and 
answers  any  questions. 

Appraisal  team 

Brings  up  any  questions  on  the  model  and 
appraisal  method. 

Table  2-11.  Participants  for  Familiarize  Team  with  SAM. 


Two  to  three  hours 


continued  on  next  page 
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2.2.2  Familiarize  Team  with  SAM,  Continued 


Steps 


Tailorable 

parameters 


Exit  criteria 


Notes 


Table  2-12  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Team  building 

The  facilitator  leads  the  team  in  a  team-building 
exercise;  at  the  very  least,  all  team  members 
introduce  themselves. 

Review  SE-CMM 

The  facilitator  presents  the  salient  features  of 
the  model  and  its  use  in  appraisal  and  discusses 
them  with  the  team. 

Review  SAM 

The  facilitator  explains  each  step  in  the 
appraisal  process.  The  team's  role  is  described 
with  particular  emphasis  on  behavior  and  note¬ 
taking  procedures  to  be  followed  during 
interviews.  The  facilitator  presents  the 
techniques  that  will  be  followed  to  manage 
data,  develop  results,  and  reach  consensus. 

Question  and  answer 

The  facilitator  responds  to  the  team's  questions 
concerning  the  model  and  the  appraisal  process. 

Table  2-12.  Steps  for  Familiarize  Team  with  SAM. 


•  Depth  of  instruction  on  model  and  SAM,  depending  on  experience  of 
team. 

•  Types  of  tools  to  use  to  support  the  appraisal. 


•  Appraisal  team  understands  use  of  SE-CMM  in  appraisal. 

•  Appraisal  team  understands  its  role  in  appraisal  activities. 

•  Appraisal  team  commits  to  performing  the  appraisal  as  structured  in 
preparation  phase. 


Because  there  is  no  formal  training  available  for  v  1.0  of  SAM,  this  step 
is  important  to  ensure  that  the  appraisal  team  understands  the  basic  flow 
of  the  appraisal  activities  and  their  responsibilities  throughout  the  week. 
See  Appendix  E  for  some  support  materials  useful  in  familiarizing  the 
appraisal  team  with  SAM.  Other  appendices  contain  support  materials 
(e.g.,  data  tracking  sheet)  that  the  team  may  wish  to  use. 
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2.2.3  Analyze  Questionnaire 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.3  Analyze  Questionnaire  (AQ) 


The  purpose  of  Analyze  Questionnaire  is  to  develop  a  set  of  exploratory 
questions  for  use  in  the  interviews  with  the  systems  engineering  leads. 


The  appraisal  team  analyzes  the  responses  to  the  questionnaire  to 
determine  areas  for  practice  validation  and  potential  discrepancy,  and  to 
perform  a  gap  analysis  against  the  SE-CMM.  If  not  completed 
previously,  the  questionnaire  responses  are  transcribed  to  the  data 
tracking  sheets.  This  process  element  results  in  a  set  of  candidate 
exploratory  questions  (EQ)  for  the  interview  with  the  systems 
engineering  leads,  and  "listen  fors"  at  practitioner  sessions.  Training 
materials,  or  project-specific  data  such  as  plans,  meeting  agendas,  or 
action  item  lists,  that  have  been  collected  may  be  reviewed  and  used  to 
support  EQ  development  or  corroborate  questionnaire  answers. 


Table  2-13  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Provides  guidance  in  the  formulation  of 
exploratory  questions  and,  if  necessary, 
transcription  of  the  questionnaire 
responses  to  the  DTS. 

Appraisal  team 

Develops  and  agrees  upon  set  of  follow-up 
questions  for  each  of  the  SE  leaders,  and 
"listen  fors"  during  practitioner  sessions. 

Table  2-13.  Participants  for  Analyze  Questionnaire. 


Four  hours 


continued  on  next  page 
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2.2.3  Analyze  Questionnaire,  continued 


Steps 


Table  2-14  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Review  responses 

The  appraisal  team  reviews  the  questionnaires 
and  the  data  tracking  sheets. 

Transcribe  to  DTS 
(if  necessary) 

The  appraisal  team,  with  guidance  from  the 
facilitator,  transcribes  the  questionnaire 
responses  from  the  original  questionnaires  to 
the  DTS.  This  is  most  easily  accomplished 
with  2-person  teams,  with  one  calling  out 
question  number  and  response,  and  the  other 
recording  in  the  appropriate  spot  in  the  DTS. 
With  a  6-person  appraisal  team  (three  2-person 
teams),  all  17  process  areas  can  be  fairly  easily 
transcribed  in  an  hour  (assuming  3-4 
respondents  per  questionnaire).  The 
handwritten  DTSs  can  then  be  transcribed  by 
support  staff  for  future  electronic  processing. 

Generate  questions 

The  appraisal  team,  with  guidance  from  the 
facilitator,  generates  20-40  exploratory 
questions  for  each  systems  engineering  lead. 

The  questions  should  be  designed  to  elicit  more 
then  just  a  yes/no  response.  For  example, 
questions  often  begin  with,  “Would  you  please 
describe  . . When  looking  for  certain 
responses,  e.g.,  “SE  management  plan,”  note 
these  words  as  “listen  fors"  that  the  facilitator 
can  use  as  a  cue  to  ask  additional  questions  if 
they  are  not  mentioned.  Some  questions  may 
be  accompanied  by  a  request  for  relevant  or 
supporting  documents. 

Questions  should  be  used  to  refine  answers  or 
explore  inconsistencies.  There  is  a  separate  set 
of  questions  for  each  systems  engineering  lead. 
However,  there  is  usually  some  overlap.  Once 
the  questions  are  developed  they  should  be 
transcribed  into  an  appropriate  form,  and  copies 
should  be  made  for  each  team  member.  The 
form  should  list  each  question,  any  “listen 
fors,"  document  requests,  and  room  for  notes. 

Table  2-14.  Steps  for  Analyze  Questionnaire. 


continued  on  next  page 
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2.2.3  Analyze  Questionnaire,  Continued 


Steps,  continued 


Step 

Guidance 

Generate  questions, 
continued 

If  the  exploratory  question  generation  process 
is  not  well  managed,  it  can  easily  exceed  the 
typical  four  hour  duration.  A  method  for 
quickly  generating  a  pool  of  questions  follows: 

•  Facilitator  leads  exploratory  question 
generation  example  with  one  process  area  to 
get  team  oriented  to  using  the  data  embedded 
in  the  questionnaire. 

•  Process  areas  are  split  up  between  the 
appraisal  team  members  according  to  their 
specialty  areas  and  to  provide  balance  in 
workload. 

•  For  each  process  area,  the  team  member  uses 
the  initial  DTS  to  scan  for  inconsistencies 
between  the  projects  and  within  a  project. 

•  Where  probe  points  are  discovered,  the  team 
member  formulates  a  candidate  question  on  a 
post-it  note,  marked  with  team  member’s 
initials,  the  SE  lead  (or  leads)  for  whom 
question  is  intended,  and  the  PA  item 
reference  that  led  to  the  question. 

•  The  post-its  can  be  laid  out  in  a  matrix  with 
PAs  across  the  top  and  SE  leads  down  the 
side  to  get  a  picture  of  the  sampling  space 
represented  by  the  initial  question  pool. 

•  The  team  reconvenes  and  comes  to  consensus 
on  which  items  to  ask  each  SE  lead. 

Table  2-14.  Steps  for  Analyze  Questionnaire,  continued 


•  When  analysis  is  performed  (may  be  done  before  on-site  week). 


•  Exploratory  questions  and  data  requests  are  prepared  for  each  systems 
engineering  lead. 


The  facilitator  can  prepare  for  this  step  before  the  on-site  period.  A  team 
may  request  and  use  documentation  generated  for  or  by  a  process  area  to 
help  demonstrate  a  capability  level,  in  addition  to  the  interview  data. 


Tailorable 

parameters 

Exit  criteria 

Notes 
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2.2.4  interview  Systems  Engineering  Leads 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


2.2.4  Interview  Systems  Engineering  Leads  (IL) 


The  purpose  of  Interview  Systems  Engineering  Leads  is  to  resolve  any 
misunderstandings  about  the  responses  to  the  questionnaire,  and  to 
explore  areas  that  the  appraisal  team  wishes  to  have  clarified. 


The  session  facilitator  introduces  the  team,  explains  the  purpose  of  the 
session,  and  asks  the  systems  engineering  lead  the  exploratory 
questions.  A  session  recorder  tracks  the  responses  and,  along  with  the 
rest  of  the  appraisal  team,  takes  notes.  As  a  result  of  some  responses, 
the  systems  engineering  lead  may  be  asked  to  supply  certain  documents 
for  later  review.  There  is  a  separate  session  for  each  systems 
engineering  lead. 


Table  2-15  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Session  facilitator 

Conducts  session  and  asks  exploratory 
questions.  The  session  facilitator  need  not  be 
the  appraisal  facilitator.  If  team  members 
possess  the  skills  and  experience  to  conduct 
structured  interviews,  they  should  be 
encouraged  to  conduct  at  least  some  of  the 
interviews  with  the  systems  engineering  leads. 

Session  recorder 

Records  responses,  monitors  the  time,  and 
makes  a  list  of  any  requested  documentation. 

Appraisal  team 

Takes  notes  of  responses  and  occasionally  asks 
for  additional  clarification. 

Systems  engineering 
lead 

Responds  to  questions. 

Table  2-15.  Participants  for  Interview  Systems  Engineering 

Leads. 


continued  on  next  page 


SECMM-94-06ICMU/SEi-96-HB-004  v1 .1 


2-27 


2.2.4  Interview  Systems  Engineering  Leads,  Continued 


Typical  duration 

Steps 


Tailorable 

parameters 

Exit  criteria 


One  hour  per  session  +15  minutes  for  team  discussion  and  a  15  minute 
break. 


Table  2-16  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Introduction 

The  session  facilitator  welcomes  the  systems 
engineering  lead  and  introduces  the  appraisal 
team.  The  facilitator  then  explains  the  purpose 
of  the  session  and  how  it  will  be  conducted. 
He/she  also  reminds  the  systems  engineering 
lead  about  the  confidentiality  mles  and 
encourages  frank  responses  to  the  questions. 

Exploratory 

questions 

The  session  facilitator  asks  each  exploratory 
question  while  the  session  recorder  keeps  track 
of  the  time  and  any  requests  for  documents. 

All  other  team  members  take  notes.  Any  team 
member  can  ask  a  question,  though  the  session 
facilitator  should  be  allowed  to  lead  the 
questioning  to  ensure  that  all  high-priority 
questions  are  covered.  The  session  facilitator 
should  follow-up  on  the  “listen  fors"  and  make 
any  document  requests. 

Closing 

At  the  end  of  the  questions  (or  when  time  has 
expired),  the  session  facilitator  concludes  the 
session  by  reminding  the  systems  engineering 
lead  of  the  time  and  location  of  their  next 
meeting  for  reviewing  the  preliminary  findings. 
The  session  recorder  reminds  the  systems 
engineering  lead  of  any  document  requests  and 
arranges  for  collecting  them.  The  session 
facilitator  thanks  the  systems  engineering  lead 
for  his/her  cooperation. 

Table  2-16.  Steps  for  Interview  Systems  Engineering  Leads. 


•  Who  is  assigned  as  the  session  facilitator. 

•  Depth  of  follow-up  on  individual  issues. 


•  As  many  exploratory  questions  as  possible  are  covered. 


continued  on  next  page 


2-28 


SECMM-94-06ICMU/SEI-96-HB-004  v1 .1 


2.2.4  Interview  Systems  Engineering  Leads,  continued 


Notes 


The  reason  exploratory  questions  are  ordered  by  priority  is  to  make  sure 
that  the  most  important  questions  (to  the  team)  are  asked.  It  is 
uncommon  for  all  the  scripted  questions  to  be  addressed.  A  team  may 
request  and  use  documentation  generated  for  or  by  a  process  area  to 
demonstrate  a  capability  level. 

Depending  on  the  acceptability  in  the  appraisal  organization's  culture,  it 
is  very  useful  to  have  the  session  recorder  use  a  laptop  or  other 
workstation  to  key  in  responses  and  dialogue  from  EQs  as  they  are 
happening.  If  the  EQs  have  been  prepared  online,  and  responses  are 
recorded  in  spaces  between,  this  permits  much  more  rapid  consolidation 
of  post-interviews.  More  than  one  or  two  laptops  being  used  seems 
daunting  to  most  interviewees,  however,  and  manual  note  taking  is 
recommended  for  most  team  members.  Ask  before  using  a  laptop  or 
other  recording  instrument. 
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2.2.5  Consolidate  Data  from  SE  Leads 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.5  Consolidate  Data  from  SE  Leads  (CDl) 


The  purpose  of  Consolidate  Data  from  SE  Leads  is  to  assimilate  the 
notes  taken  during  the  interviews  with  the  systems  engineering  leads. 


Consolidate  Data  from  SE  Leads  involves  updating  the  DTS  to  reflect 
the  information  gained  in  the  interviews  with  the  systems  engineering 
leads.  It  also  allows  the  members  of  the  appraisal  team  to  verify  their 
understanding  of  the  information  obtained  in  the  interviews  with  the 
other  team  members.  Finally,  this  data  consolidation  step  allows  the 
team  to  strategize  any  needed  changes  in  the  scheduling  or  other  aspects 
of  the  remaining  data  gathering. 


Table  2-17  lists  the  primary  roles  involved  in  this  process  element  and 
the  summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Provides  guidance  and  model  expertise  for  the 
team's  deliberations. 

Appraisal  team 

Reviews  their  notes,  discusses  any  issues, 
formulates  observations,  and  updates  their  data 
tracking  sheets 

Table  2-17.  Participants  for  Consolidate  Data  from  SE  Leads. 


One  hour 


continued  on  next  page 
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2.2.5  Consolidate  Data  from  SE  Leads,  continued 


Steps 


Tailorable 

parameters 

Exit  criteria 

Notes 


Table  2-18  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Review  notes 

Each  team  member  privately  reviews  his/her 
notes  from  the  interviews  with  the  systems 
engineering  leads. 

Update  DTS 

Each  team  member  updates  the  data  tracking 
sheets  by  adding  a  plus  (+)  or  minus  (-)  to 
those  base/generic  practices  for  which 
corroborating  or  opposing  evidence  was  heard. 

Discuss  issues 

Team  miembers  raise  issues  that  they  have 
encountered  during  the  review  and  update 
steps.  The  facilitator  moderates  the  discussions 
to  give  each  issue  a  chance  to  be  aired.  At  the 
end  of  the  discussion,  team  members  may  wish 
to  alter  their  entries  to  their  data  tracking  sheets. 

Preparation 

The  team  discusses  the  upcoming  events  and 
any  alterations  that  they  may  wish  to  make 
based  on  the  previous  discussions.  In 
particular,  the  team  may  want  the  session 
facilitators  to  guide  the  discussions  into  certain 
areas  during  one  or  more  of  the  practitioner 
interviews.  This  direction  should  be  carefully 
limited,  as  too  much  direction  of  the  practitioner 
discussions  will  stifle  spontaneity. 

Table  2-18.  Steps  for  Consolidate  Data  from  SE  Leads. 


•  None 


•  Team  members  complete  update  of  their  DTS. 

•  Changes  to  schedule  and  other  data  gathering  mechanisms  are  agreed 
to. 


See  Appendix  C  for  details  on  the  format  and  suggested  use  of  the  data 
tracking  sheets. 
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2.2.6  Interview  Practitioners 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.6  Interview  Practitioners  (IP) 


The  purpose  of  Interview  Practitioners  is  to  meet  with  the  actual  systems 
engineers  and  practitioners  of  related  support  processes  (e.g.,  quality 
assurance,  manufacturing)  to  obtain  corroboration  of  the  key  issues 
previously  asserted  and  to  identify  new  issues. 


Interview  Practitioners  involves  facilitating  a  relatively  free-form 
discussion  centered  around  the  question,  "What  works  or  doesn’t  work 
well  in  the  systems  engineering  process?"  This  discussion  typically 
provides  corroborating  and  clarifying  data  in  relation  to  other  sources. 

In  some  cases,  the  facilitator  may  provide  minimal  direction  into  topic 
areas  needed  to  be  “filled  in,”  based  on  the  DTS.  It  is  usually  best  to  let 
the  free-form  part  of  the  discussion  begin  the  session;  otherwise, 
participants  may  get  the  impression  of  a  question  and  answer  format, 
and  cease  volunteering  the  issues  that  are  important  to  them. 


Table  2-19  lists  the  prim^  roles  involved  in  this  process  element  and 
the  summary  of  their  activity  during  this  process. 


Role 

Summary 

Session  facilitator 

Keeps  discussion  moving  without  influencing  its 
direction. 

Appraisal  team 

Takes  notes  of  discussions. 

Practitioners 

Present  issues  from  their  perspective. 

Table  2-19.  Participants  for  Interview  Practitioners. 


Two  hours  per  practitioner  group 


continued  on  next  page 
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2.2.6  Interview  Practitioners,  continued 


Steps 


Table  2-20  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Preparation 

Arrange  the  interview  area  before  the  practitioners’ 
arrival.  The  setup  can  be  around  a  table  or  in  a 
circle  without  a  table.  In  either  case,  the  team 
should  occupy  every  other  seat,  forcing  the 
practitioners  to  spread  out  around  the  circle.  The 
team  members  should  be  in  their  seats  before  the 
practitioners  arrive.  It  is  important  to  begin  the 
meeting  on  time.  If  possible,  have  someone  not  on 
the  team  call  any  latecomers  or  no-shows. 

Introduction 

The  session  facilitator 

•  Welcomes  the  practitioners  and  introduces  the 
session’s  topic  (if  appropriate). 

•  Explains  the  purpose  of  the  session  and  how  it 
will  be  conducted,  i.e.,  an  unstructured 
discussion  of  the  systems  engineering  process 
from  the  viewpoint  of  the  practitioners. 

•  Lets  the  practitioners  know  that  the  team  will 
listen  and  take  notes  but  that  the  direction  of  the 
discussions  will  be  up  to  them. 

•  Warns  them  that,  at  the  end  of  the  session,  each 
participant  will  be  asked  for  a  strength  and  a 
weakness  of  the  appraised  entity. 

The  session  facilitator  reminds  the  practitioners 
about  the  confidentiality  rules,  encourages  frank 
discussions,  and  cautions  the  participants  about 
repeating  comments  outside  of  the  interview.  The 
team  and  practitioners  then  introduce  themselves, 
giving  their  name  and  role  (team  members  should 
identify  themselves  as  "members  of  the  appraisal 
team’’).  The  facilitator  then  turns  the  meeting  over 
to  the  practitioners. 

Table  2-20.  Steps  for  Interview  Practitioners. 


continued  on  next  page 
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2.2.6  Interview  Practitioners,  Continued 


Steps,  continued 


Step 

Guidance 

Discussion 

The  session  facilitator  keeps  the  conversation 
moving  with  as  little  intrusion  as  possible.  Team 
members  should  not  participate  in  the 
conversations.  The  session  facilitator  should 
note  non-participating  practitioners  and  try  to 
bring  them  into  the  conversation  in  a  non-directed 
manner  (e.g.,  "What  do  you  think  about  that, 
Tom?"). 

If  the  discussions  lag,  or  special  topics  have  been 
identified,  the  session  facilitator  should  gently 
"nudge"  the  discussion  in  the  desired  direction. 

If  this  is  not  done  carefully,  the  participants  will 
continually  look  to  the  facilitator  for  direction  and 
the  interview  will  loose  its  spontaneity.  The 
facilitator  should  use  leading  comments  (e.g., 
"How  about  quality  assurance  . . .?"),  and  avoid 
direct  yes/no  questions  (e.g.,  "Do  you  do  quality 
assurance?"),  or  judgmental  questions  (e.g., 

"Why  don’t  you  do  quality  assurance?"). 

Table  2-20.  Steps  for  Interview  Practitioners,  continued 


continued  on  next  page 
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2.2.6  Interview  Practitioners,  continued 


Steps,  continued 


Tailorable 

parameters 


Exit  criteria 


Step 

Guidance 

Closing 

Near  the  end  of  the  session,  the  facilitator  brings 
the  conversation  to  an  end  and  asks  two 
questions  of  each  practitioner.  If  the  facilitator 
senses  that  the  discussions  have  reached  the  end 
of  their  usefulness,  he/she  can  end  the  session 
before  the  end  of  the  full  time  period. 

The  session  facilitator  then  asks  each  practitioner 
the  following  question: 

If  you  could  change  one  thing  in  your 
organization  other  than  your  boss  or  your 
paycheck,  what  would  it  be  ? 

Next,  the  facilitator  asks  each  practitioner 
(perhaps  going  around  the  group  in  the  opposite 
direction): 

Other  than  the  people,  what  do  you  think 
is  this  organization's  major  strength? 

Note  the  word  "organization"  should  be 
customized  for  the  situation. 

Finally,  the  session  facilitator  thanks  the 
practitioners  for  their  participation  and  reminds 
them  of  the  time  and  location  for  presenting  the 
draft  findings. 

Process  check 

In  the  interval  between  practitioner  interviews, 
the  team  should  perform  a  brief  process  check  of 
how  the  last  interview  went,  and  discuss  any 
changes  or  special  situations  anticipated  with  the 
next  group.  The  sessions  are  long,  so  be  sure  to 
leave  time  for  breaks  before  the  next  group 
arrives. 

Table  2-20.  Steps  for  Interview  Practitioners,  continued 


•  Number  and  makeup  of  practitioner  groups. 

•  Use  of  middle  manager  or  other  focus  groups,  perhaps  outside  of  on¬ 
site  week. 


•  Session  completed. 


continued  on  next  page 
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2.2.6  Interview  Practitioners,  Continued 


Notes 


Practitioner  groups  may  be  grouped  according  to  life-cycle  phase 
(requirements  engineers/architects  in  one  group,  integrators/testers  in 
another),  organizational  boundaries  (all  the  CM  and  training  folks 
together  since  they  are  outside  the  systems  engineering  organization 
hierarchy),  or  other  ways  that  will  maximize  the  information  obtained. 

A  project  management  practitioner  group  might  be  set  up  if  the  systems 
engineering  and  project  management  functions  (as  defined  in  the  project 
section  of  the  SE-CMM)  are  across  organizational  boundaries.  The 
point  is  to  use  the  particular  organizational  context  to  best  advantage  to 
corroborate  data  obtained  via  the  questionnaire  and  interviews  with  the 
systems  engineering  leads.  A  team  may  request  and  use  documentation 
generated  for  or  by  a  process  area  to  demonstrate  a  capability  level. 

Middle  manger  buy-in  may  be  obtained  by  having  a  practitioner  or  focus 
group  before  the  on-site  week,  which  provides  an  opportunity  to  engage 
them  as  "owners"  of  the  action  plan  recommendations  after  the  on-site 
period.  In  addition,  in  some  organizations,  marketing  or  business 
development  will  "own"  product  line  evolution,  and  should  be  included 
in  practitioner  groups  or  have  a  mini-focus  group. 

A  session  recorder  may  use  a  laptop  or  other  workstation  if  the 
organization's  culture  permits  it.  This  greatly  facilitates  consolidation  of 
data  and  application  of  key  phrases  to  findings.  Ask  before  using  a 
laptop  or  other  recording  equipment.  In  no  case  should  more  than  one 
or  two  use  laptops. 
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2.2.7  Consolidate  Data  from  Practitioners 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


12.1  Consolidate  Data  from  Practitioners  (CD2) 


The  purpose  of  Consolidate  Data  from  Practitioners  is  to  assimilate  the 
notes  taken  during  the  practitioner  interviews  and  form  preliminary 
ratings  for  each  process  area. 


Consolidate  Data  from  Practitioners  involves  updating  the  DTS  to  reflect 
the  information  gained  in  the  interviews  with  the  systems  engineering 
leads.  It  also  allows  the  members  of  the  appraisal  team  to  verify  their 
understanding  of  the  information  obtained  in  the  interviews  with  the 
other  team  members.  Finally,  this  data  consolidation  step  allows  the 
team  to  strategize  any  needed  changes  in  the  scheduling  or  other  aspects 
of  the  remaining  data  gathering.  In  addition,  Consolidate  Data  from 
Practitioners  is  the  first  time  when  the  team  attempts  to  develop  a  rating 
profile. 


Table  2-21  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Provides  guidance  and  model  expertise  for  the 
team's  deliberations. 

Appraisal  team 

Review  their  notes,  discuss  any  issues,  formulate 
observations,  update  their  data  tracking  sheets, 
and  formulate  preliminary  ratings. 

Table  2-21.  Participants  for  Consolidate  Data  from  Practitioners. 


One  hour 


continued  on  next  page 
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2.2.7  Consolidate  Data  from  Practitioners,  Continued 


Step 


Table  2-22  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Review  notes 

Each  team  member  privately  reviews  his/her 
notes  from  the  practitioner  interviews, 
committing  issues  to  post-its. 

Update  DTS 

Each  team  member  updates  his/her  data  tracking 
sheets  by  adding  a  plus  (+)  or  minus  (-)  to 
those  base/generic  practices  for  which 
corroborating  or  opposing  evidence  was  heard. 

Discuss  issues 

Team  members  raise  issues  that  they  have 
encountered  during  the  review  and  update 
steps.  The  facilitator  moderates  the  discussions 
to  give  each  issue  a  chance  to  be  aired.  At  the 
end  of  the  discussion,  team  members  may  wish 
to  alter  their  entries  to  their  data  tracking  sheets. 

Form  preliminary 
ratings 

Based  on  their  data  tracking  sheets,  the 
appraisal  team  forms  a  preliminary  rating  for 
each  process  area.  A  capability  level  is 
considered  achieved  if,  in  the  opinion  of  the 
team  members,  100%  of  the  base/generic 
practices  of  that  level  are  performed.  To 
achieve  level  1,  90%  of  the  base  practices, 
across  the  appraisal  entity  as  a  whole,  must  be 
performed.  For  levels  2  through  5,  50%  of  the 
generic  practices  associated  with  that  level  must 
be  performed  across  the  organization. 

It  is  not  critical  that  the  team  achieve  consensus 
at  this  time;  however,  plans  must  be  made  for 
resolving  any  conflicts.  In  this  case, 
preliminary  findings  can  be  crafted  to  explore 
any  open  issues  further. 

Preparation 

The  team  discusses  the  upcoming  events  and 
any  alterations  that  they  may  wish  to  make 
based  on  the  previous  discussions.  In 
particular,  the  team  may  wish  to  add  a  few 
exploratory  questions  or  special  findings  to  the 
preliminary  findings  in  order  to  resolve 
conflicting  evidence,  especially  where  the 
ratings  are  affected. 

Table  2-22.  Steps  for  Consolidate  Data  from  Practitioners. 


continued  on  next  page 
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2.2.7  Consolidate  Data  from  Practitioners,  continued 


Tailorable 

parameters 

Exit  criteria 

Notes 


•  Initial  rating  profile  is  developed  (may  be  deferred). 


•  DTS  are  updated. 

•  Scheduling  and  data  gathering  changes  are  agreed  to  where 
appropriate. 


See  Appendix  C  for  details  on  the  format  and  suggested  use  of  the  data 
tracking  sheets. 

It  is  recommended  that  each  team  member  copy  issues  (strengths  and 
weaknesses)  to  post-its  as  notes  are  reviewed.  The  affected  PA  and 
team  member's  initials  should  be  included  together  with  the  session  in 
which  the  issue  was  recorded.  A  typical  team  member  generates  30- 
100+  issues,  and  can  generate  250-400  total. 
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2.2.8  Develop  Preliminary  Findings 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


2.2.8  Develop  Preliminary  Findings  (PF) 


The  purpose  of  Develop  Preliminary  Findings  is  to  formulate  a  set  of 
findings  that  reflect  an  initial  synthesis  of  the  accumulated  data  from  all 
data  sources  used  in  the  appraisal. 


During  Develop  Preliminary  Findings,  the  appraisal  team  systematically 
analyzes  the  data  from  all  sources  to  generate  a  list  of  preliminary 
findings  related  to  the  process  areas  and  capability  levels  under 
investigation. 


Table  2-23  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Leads  team  in  brainstorm  activities  and 
oversees  the  preparation  of  the  preliminary 
findings. 

Appraisal  team 

Synthesize  the  accumulated  data  into 
preliminary  findings,  including  both  strengths 
and  weaknesses. 

Table  2-23.  Participants  for  Develop  Preliminary  Findings. 


continued  on  next  page 
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2.2.8  Develop  Preliminary  Findings,  Continued 


Steps 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 

Notes 


Table  2-24  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Individual  appraisal 
team  members  record 
their  candidate 
preliminary  findings 
according  to  process 
areas/common 
features 

An  easy  way  to  record  candidate  findings  is  to 
record  them  on  post-it  notes  along  with  the 
process  area  or  common  feature  tag  and  author 
initials.  You  can  show  their  affinities  by 
placing  the  notes  either  on  a  long  table  or  on 
multiple  flip  charts  (one  per  PA). 

Mini-teams  collate  the 
findings,  eliminate 
redundancies,  and 
look  for  common 
threads 

Teams  of  two  work  well  to  narrow  the  initial 
set  by  clustering  to  a  number  that  the  entire 
team  can  review.  The  teams  can  move  to  the 
next  process  area  in  line  after  they  are  done 
with  their  first  one,  and  so  on  until  all  are 
clustered. 

Mini-teams  present 
candidates  to  entire 
team  for  consensus  on 
findings  to  put  forth 

Consensus  here  is  more  of  the  nature  of,  "Can  I 
live  with  the  systems  engineering  team  leads' 
agreement  or  disagreement?"  rather  than  "I  can 
agree  with  this."  Remember,  not  all  the 
findings  will  be  validated  by  the  feedback 
sessions. 

Preliminary  findings 
are  recorded  for  use  in 
the  feedback  sessions 
for  the  next  day 

We  recommend  that  you  use  a  flat  file  database, 
such  as  a  word  processor,  to  record  the 
findings  according  to  process  area/common 
feature  with  a  form  that  allows  room  for  notes 
from  the  feedback  session.  This  makes  it  easy 
for  the  team  to  record  notes  the  following  day. 

Table  2-24.  Steps  for  Develop  Preliminary  Findings. 


Three  to  six  hours 


•  Level  of  granularity  of  preliminary  findings. 

•  Number  of  preliminary  findings  presented  for  validation. 


•  40-60  preliminary  findings  that  can  be  verified  with  the  systems 
engineering  leads. 


One  PA  per  flipchart  is  recommended  with  an  additional  flipchart  each 
for  "strengths"  and  "not  in  model"  issues. 
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2.2.9  Review  Preliminary  Findings 


Element  title/tag 
Purpose 

Summary 

description 

Major 

participants 


Typical  duration 


2.2.9  Review  Preliminary  Findings 


The  purpose  of  Review  Preliminary  Findings  is  to  obtain  feedback  on 
the  preliminary  findings  from  the  systems  engineering  leads  who 
provided  the  original  data. 


Review  Preliminary  Findings  involves  feeding  back  the  synthesized  data 
in  the  form  of  strengths  and  weaknesses  to  the  same  systems 
engineering  leads  who  were  the  original  sources  of  the  data. 


Table  2-25  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Session  facilitator 

Presents  preliminary  findings  to  the  systems 
engineering  leads  for  their  comment.  If 
possible,  the  session  facilitator  role  should  be 
filled  by  the  appraisal  team  leader  to  increase 
his/her  comfort  with  explaining  the  findings. 

Session  recorder 

Tracks  session  and  monitors  time.  If  the  team 
leader  is  the  session  facilitator,  the  trained 
facilitator  should  act  as  the  session  recorder. 

Systems  engineering 
lead 

Provides  feedback  on  preliminary  conclusions. 

Appraisal  team 

Takes  notes. 

Table  2-25.  Participants  for  Review  Preliminary  Findings. 


One  hour  per  project 


continued  on  next  page 
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2.2.9  Review  Preliminary  Findings,  Continued 


Steps 


Table  2-26  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Introduction 

The  session  facilitator  welcomes  the  systems 
engineering  lead  and  describes  what  the  team  has 
done  since  the  last  meeting.  He/she  describes  the 
purpose  of  this  session  and  explains  that  the 
findings  are  preliminary  and  will  likely  change; 
therefore,  it  is  especially  important  not  to  discuss 
this  version  of  the  findings  outside  of  this  review. 

The  facilitator  also  tells  the  lead  that  at  the  end  of 
the  session  he/she  will  ask  for  a  strength  and  an 
important  change  in  the  appraised  entity. 

Review  findings 

The  session  facilitator  presents  each  preliminary 
finding  to  the  systems  engineering  lead  and  asks 
whether  he/she  believes  that  the  finding  is  tme  on 
their  project  and  whether  he/she  believes  it  is 
generally  true  for  the  appraised  entity. 

The  facilitator  or  members  of  the  team  can  ask  for 
clarification  or  a  follow-on  question.  However, 
all  preliminary  findings  need  to  be  presented  in  the 
allocated  time.  Therefore,  the  facilitator  and 
recorder  need  to  keep  a  tight  rein  on  the  time  and 
not  let  follow-up  questions  jeopardize  the 
schedule. 

Table  2-26.  Steps  for  Review  Preliminary  Findings. 


continued  on  next  page 
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2.2.9  Review  Preliminary  Findings,  Continued 


Steps,  continued 


Step 

Guidance 

Closing 

The  session  facilitator  closes  the  session  by  asking 
the  systems  engineering  lead  the  following 
question: 

If  you  could  change  one  thing  in  your 
organization  other  than  your  boss  or  your 
paycheck,  what  would  it  be? 

Next  the  facilitator  asks: 

Other  than  the  people,  what  do  you  think  is 
this  organization's  major  strength? 

Note  the  word  "organization"  should  be 
customized  for  the  situation. 

Finally,  the  session  facilitator  thanks  the  systems 
engineering  lead  for  participating  and  reminds 
him/her  of  the  time  and  location  for  presenting  the 
draft  findings. 

Process  check 

In  the  interval  between  interviews,  the  team  should 
perform  a  brief  process  check  of  how  the  last 
interview  went,  and  discuss  any  changes  or 
special  situations  anticipated  with  the  next  systems 
engineering  lead.  Again,  be  sure  to  leave  time  for 
a  visit  to  the  bathroom  and  back  before  the  next 
lead  arrives. 

Table  2-26.  Steps  for  Review  Preliminary  Findings,  continued 


Tailorable 

parameters 


Documentation  may  be  requested  and  reviewed. 

SE  leads  may  be  provided  a  table  of  preliminary  findings  and  check 
yes/no  for  project  and  organization  on  paper;  the  session  facilitator 
then  asks  if  there  are  any  questions. 

SE  leads  may  review  preliminary  findings  in  parallel  sessions, 
providing  teams  of  two  (facilitator  and  recorder)  attend  each. 


Exit  criteria 


•  Feedback  on  the  applicability  of  the  preliminary  findings  has  been 
obtained. 

•  Potential  modifications  to  findings  are  noted  by  the  team. 


Notes 


None. 
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2.2.10  Develop  Draft  Rating 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.10  Develop  Draft  Rating  (DR) 


The  purpose  of  Develop  Draft  Rating  is  to  assimilate  the  notes  taken 
during  the  review  of  preliminary  findings  and  to  formulate  a  draft  set  of 
ratings  for  each  process  area. 


Develop  Draft  Rating  involves  preparing  to  synthesize  the  preliminary 
findings  into  a  set  of  draft  findings  that  form  the  core  of  the  findings 
briefing  to  be  presented  to  the  appraisal  participants.  In  addition,  Ais  is 
the  point  at  which  the  appraisal  team  must  reach  consensus  on  the  draft 
rating  profile. 


Table  2-27  lists  the  primary  roles  involved  in  this  process  element  and 
the  summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Provides  guidance  and  model  expertise  for  the 
team's  deliberations. 

Appraisal  team 

Review  their  notes,  discuss  any  issues,  formulate 
observations,  update  their  data  tracking  sheets, 
and  formulate  draft  ratings. 

Table  2-27.  Participants  for  Develop  Draft  Rating. 


One  to  two  hours 


continued  on  next  page 
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2.2.10  D6V6lop  Drdft  RstinQj  Continued 


Steps 


Table  2-28  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Review  notes 

Each  team  member  privately  reviews  his/her 
notes  from  the  interview  with  the  systems 
engineering  leads. 

Update  DTS 

Each  team  member  updates  his/her  data  tracking 
sheets  by  adding  a  plus  (+)  or  minus  (-)  to 
those  base/generic  practices  for  which 
corroborating  or  opposing  evidence  was  heard. 

Discuss  issues 

Team  members  raise  issues  that  they  have 
encountered  during  the  review  step.  The 
facilitator  moderates  the  discussions  to  give 
each  issue  a  chance  to  be  aired.  At  the  end  of 
the  discussion,  team  members  may  wish  to  alter 
the  entries  on  their  data  tracking  sheets. 

Draft  ratings 

Based  on  their  data  tracking  sheets,  the 
appraisal  team  forms  a  preliminary  rating  for 
each  process  area.  A  capability  level  is 
considered  achieved  if,  in  the  opinion  of  the 
team  members,  the  base/generic  practices  for 
that  level  are  performed.  To  achieve  level  1, 
100%  of  the  base  practices  must  be  performed. 
For  levels  2-5,  the  generic  practices  associated 
with  that  level  must  be  performed  (but  80% 
may  be  an  acceptable  standard). 

At  this  point  in  the  appraisal  process,  team 
consensus  on  the  ratings  is  necessary.  The 
facilitator  leads  the  process  of  building 
consensus.  In  the  unlikely  event  that 
consensus  cannot  be  achieved,  the  team  must 
develop  a  plan  for  presenting  the  draft  findings 
that  will  allow  final  resolution  of  the  ratings, 
based  on  practitioner  and  SE  lead  input. 

Preparation 

The  team  discusses  the  upcoming  events  and 
any  alterations  that  they  may  wish  to  make 
based  on  the  previous  discussions.  In 
particular,  the  team  may  wish  to  include 
additional  slides  with  the  draft  findings 
briefing. 

Table  2-28. 

Steps  for  Develop  Draft  Rating. 

continued  on  next  page 
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2.2.10  Develop  Draft  Rating,  continued 


Tailorable 

parameters 

Exit  criteria 

Notes 


•  This  step  would  not  include  ratings  development  if  the  appraisal 
context  excludes  the  rating  profile  from  the  appraisal  results. 


•  Consensus  is  obtained  on  the  draft  rating  profile. 


See  Appendix  B  for  rating  profile  example. 
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2.2.1 1  Develop  Draft  Findings 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2. 1 1  Develop  Draft  Findings  (DF) 


The  purpose  of  Develop  Draft  Findings  is  to  focus  on  a  subset  of  the 
process  areas  and  to  develop  refined  findings  for  each  process  area 
investigated  in  the  appraisal. 


Develop  Draft  Findings  involves  analyzing  the  preliminary  conclusions 
in  light  of  the  contents  of  the  capability  levels,  determining  the  estimated 
process  capability  for  each  process  area  that  was  investigated,  and 
synthesizing  the  preliminary  findings  into  a  manageable  set  for 
presentation  to  the  sponsor.  The  preliminary  findings  are  synthesized 
from  around  45-50  findings  into  5-10  draft  findings  that  address 
findings,  data,  and  consequences. 


Table  2-29  lists  the  prim^  roles  involved  in  this  process  element  and 
the  summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Provides  expertise  on  the  SE-CMM  and  guides 
the  team  in  forming  consensus. 

Appraisal  team 

Forms  consensus  for  the  final  findings  and 
estimated  process  capability  levels. 

Table  2-29.  Participants  for  Develop  Draft  Findings. 


Four  to  eight  hours 


continued  on  next  page 
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2.2.1 1  Develop  Draft  Findings,  continued 


Steps 


Table  2-30  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Review  data 

The  team  should  review  the  draft  ratings  and 
the  systems  engineering  leads'  responses  to  the 
preliminary  findings.  This  information  should 
allow  the  team  to  identify  areas  for  findings. 

The  process  areas  are  a  good  starting  point,  but 
the  appraisal  is  not  limited  to  findings  that 
exactly  match  the  SE-CMM  process  areas. 

There  are  sometimes  local  issues  that  must  be 
addressed  if  the  appraisal  is  to  have  credibility 
with  the  participants. 

The  number  of  findings  should  be  limited  to  at 
least  five,  but  no  more  than  nine.  Too  many 
findings  will  be  discouraging  and  difficult  to 
address;  too  few  will  encourage  false 
confidence  and  not  provide  a  rich  enough  set  of 
issues  for  planning  process  improvement. 

Initially  the  team  may  identify  more  than  nine 
areas,  with  the  understanding  that  it  will  merge 
or  drop  some  later. 

Prioritize  findings 

There  are  usually  more  than  10  synthesized 
findings  to  start.  Prioritization  should  be  based 
on  business  goals,  if  available,  or  on  the  team's 
consensus  on  the  major  barriers  to 
improvement  in  the  organization. 

Wordsmith 

The  team  should  form  small  groups  (two  to 
three  people)  to  edit  and  wordsmith  the 
findings.  This  is  especially  tme  if  they  could 
not  agree  on  the  wording  during  the  previous 
step. 

The  facilitator  should  carefully  consider  the 
make-up  of  these  groups  and  the  assignment  of 
findings.  Sometimes  it  is  a  good  idea  to  place 
antagonists  in  the  same  group  and  give  them  the 
controversial  finding  to  edit.  Other  times, 
neutral  parties  should  do  the  editing.  In 
extreme  cases  both  approaches  might  be  tried. 
This  is  where  the  facilitator  makes  use  of 
his/her  teaming  skills  and  knowledge  of  the 
team. 

Table  2-30.  Steps  for  Develop  Draft  Findings. 


continued  on  next  page 
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2.2.1 1  Develop  Draft  Findings,  Continued 


Steps,  continued 


Step 

Guidance 

Form  consensus 

Finally,  each  finding  is  presented  to  the  group 
for  final  edit  and  approval.  The  team  must 
agree  with  each  finding.  If  team  members  go 
away  from  the  appraisal  without  full 
commitment  to  all  of  the  findings,  the  other 
participants  will  sense  the  lack  of  consensus 
and  interpret  it  as  a  weakness  in  the  findings. 

This  step  continues  until  consensus  is  achieved. 
If  necessary,  the  team  might  have  to  go  back  to 
the  previous  step  in  an  effort  to  come  to  an 
agreement. 

Prepare  slides 

The  findings  are  placed  on  slides  for 
presentation.  See  Appendix  B  for  a  sample  of 
the  findings  presentation  slides.  The  team 
members  should  be  given  copies  of  the  draft 
findings  for  their  notebook,  and  for  recording 
comments  from  the  draft  presentation.  Do  not 
make  copies  for  the  participants,  as  the  findings 
may  change  after  their  initial  presentation.  The 
appraisal  team  leader  should  take  a  copy  with 
him/her  in  order  to  prepare  for  the  presentation 
the  next  morning. 

Table  2-30.  Steps  for  Develop  Draft  Findings,  continued 


Tailorable  •  Level  of  granularity  of  findings, 

parameters  •  Depth  of  analysis  for  determining  capability  level. 


Exit  criteria  •  Consensus  is  obtained  on  draft  findings. 

•  Draft  findings  briefing  is  completed. 


Notes  Findings  may  be  presented  in  the  context  of  process  area  categories, 

process  areas,  base  practices,  generic  practices,  common  features,  or 
capability  levels,  depending  on  the  appraisal  goals. 
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2.2.12  Present  Draft  Findings 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2. 12  Present  Draft  Findings  (PF) 


The  purpose  of  Present  Draft  Findings  is  to  provide  a  vehicle  for  the 
systems  engineering  leads  and  practitioners  to  validate  that  the 
synthesized  findings  represent  the  information  provided  throughout  the 
on-site  phase  of  the  appraisal. 


Present  Draft  Findings  involves  the  appraisal  team  leader  (ATL) 
presenting  the  synthesized  findings  to  the  systems  engineering  leads  as  a 
group,  and  then  to  all  the  practitioners  as  a  group.  This  provides  the 
appraisal  team  leader  with  an  opportunity  to  "rehearse"  the  briefing,  and 
provides  the  appraisal  participants  with  the  opportunity  to  provide 
feedback  on  the  validity  of  the  information. 


Table  2-31  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team 
leader 

Presents  the  draft  findings;  does  not  present  the 
rating  profile  or  the  introductory  material,  just  the 
summarized  findings;  solicits  comments  from 
participants. 

Appraisal  team 

Takes  notes  on  presentation  for  feedback  to 
appraisal  team  leader;  observes  reactions  of 
participants  to  findings;  takes  notes  on  feedback. 

Appraisal 

participants 

Listen  to  findings  presentation;  provide  feedback 
on  whether  the  appraisal  team  captured  what  is 
happening  in  the  organization. 

Session  recorder 

Records  agreed-upon  changes  to  briefing  on  actual 
slides  directly,  as  well  as  on  a  hard  copy. 

Session  facilitator 

Facilitates  the  feedback  portion  of  the  meeting; 
appraisal  team  leader  may  assume  this  role  if 
desired. 

Table  2*31 .  Participants  for  Present  Draft  Findings. 


One  hour  for  systems  engineering  leads  as  a  group 
One  hour  for  all  practitioners  as  a  group 
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2.2.12  Pr©s©nt  Dr©ft  FindinQS,  Continued 


Steps 


Table  2-32  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Describe  purpose  of 
session 

The  presenter  re-emphasizes  that  the  goal  of  the 
appraisal  is  to  capture  the  "state  of  the  practice" 
in  the  appraised  entity;  this  session  allows  the 
appraisal  team  to  validate  that  they  have 
accurately  captured  the  practitioners'  viewpoint. 
The  presenter  then  asks  to  go  through  entire 
briefing  and  then  come  back  to  questions.  It  is 
not  recommended  to  pass  out  a  copy  of  the 
briefing  at  this  time;  it  is  subject  to  change,  and 
incorrect  versions  of  the  briefing  could  damage 
the  credibility  of  the  appraisal. 

Present  findings 

Appraisal  team  leader  rehearses  his/her 
presentation  of  the  findings  portion  of  the 
briefing.  Not  showing  the  rating  profile  also 
gives  participants  another  reason  to  attend  the 
main  findings  briefing.  During  this  briefing, 
the  presenter  tries  to  notice  the  reactions  he/she 
gets  from  different  findings. 

Solicit  feedback 

After  the  run-through  of  the  findings,  the 
appraisal  team  leader  solicits  feedback  from  the 
practitioners,  e.g.,  "Is  this  what  you  told  us 
and  is  this  worded  in  a  way  to  get  positive 
action  from  management?"  The  ATL  facilitates 
the  discussion,  and  when  changes  are  proposed 
by  a  member  of  the  practitioner's  group,  the 

ATL  tries  to  get  a  sense  of  the  agreement  within 
the  group;  there  will  not  be  time  to  get  a  full 
consensus,  the  ATL  will  just  try  to  make  sure 
that  the  comment  is  one  Aat  engenders  general 
agreement.  The  appraisal  team  is  not  looking 
for  a  lot  of  changes,  but  is  just  making  sure  that 
the  tone  of  the  findings  is  able  to  be  acted  upon, 
and  that  the  team  is  not  misrepresenting  the  data 
that  were  gathered. 

Table  2-32.  Steps  for  Present  Draft  Findings. 


continued  on  next  page 
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2.2.12  Present  Draft  Findings,  Continued 


Steps,  continued 


Tailorable 

parameters 


Step 

Guidance 

Make  draft  changes 

Draft  changes  can  be  made  directly  on  the  actual 
slide,  so  participants  can  see  the  direct  effect  of 
their  input;  they  should  also  be  recorded  on  a 
hard  copy  or  electronic  copy  for  discussion 
among  the  team.  (These  draft  changes  will  be 
validated  in  the  next  process  element.) 

Provide  reminders 

The  ATL  reminds  participants  of  the  time  for 
the  findings  briefing  and  emphasizes  that  this  is 
an  opportunity  to  interact  with  the  management 
who  sponsored  the  appraisal.  The  ATL  asks 
the  practitioners  to  keep  findings  to  themselves 
until  after  the  findings  briefing.  This  gives 
sponsors  a  chance  to  respond  to  the  findings 
without  the  "rumor  mill"  interfering. 

Adjourn  meeting 

The  ATL  thanks  participants  for  their 
participation  throughout  the  week. 

Table  2*32.  Steps  for  Present  Draft  Findings,  continued 


•  Systems  engineering  lead  and  practitioner  groups  could  conceivably 
be  combined,  though  this  risks  inhibition  of  the  practitioners  in  some 
environments.  If  a  management  practitioner  group  was  used,  you 
may  want  to  combine  them  with  the  systems  engineering  leads. 

•  Someone  other  than  the  appraisal  team  leader  may  be  selected  as  the 
presenter  for  the  findings  briefing,  or  the  presentation  may  be  split 
between  multiple  presenters.  A  common  split  is  to  have  the  facilitator 
present  the  introduction,  the  appraisal  team  leader  present  the  findings, 
and  the  facilitator  close  with  the  "next  steps."  Presentation  by  the 
appraisal  team  leader  has  the  advantage  of  organizational  "ownership" 
of  the  material;  in  some  environments,  the  facilitator  has  the  advantage 
of  being  the  outside  "independent"  expert.  The  decision  of  who 
presents  the  findings  should  be  based  on  the  team's  consensus  of  who 
will  have  the  most  impact  and  achieve  the  most  momentum  for 
improvement  action. 
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2.2.12  Present  Draft  Findings,  continued 


Exit  Criteria 


Notes 


•  Presentation  of  draft  findings  complete. 

•  Practitioner  concerns  recorded. 


This  session  is  important  in  establishing  the  credibility  of  the  appraisal 
with  the  practitioners  and  for  providing  practitioners  with  momentum 
toward  change.  It  is  important  to  make  sure  there  are  no  findings  where 
the  systems  engineering  leads/practitioners  say,  "This  is  absolutely  not 
true  —  whatever  gave  you  that  idea?"  or  something  similar.  There 
should  not  be  many  changes  that  come  out  of  these  presentations,  but 
you  should  put  a  couple  in,  even  if  they  are  minor  details,  just  to  show 
the  team's  willingness  to  listen. 
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2.2.13  Adjust  Draft  Findings 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


2.2.13  Adjust  Draft  Findings  (AD) 


The  purpose  of  Adjust  Draft  Findings  is  to  ensure  that  the  final  findings 
briefing  accurately  reflects  the  information  obtained  from  the  participants 
by  refining  the  findings  briefing  based  on  feedback  from  the  appraisal 
participants. 


Adjust  Draft  Findings  involves  the  appraisal  team  discussing  the 
presentation  of  the  draft  findings  to  the  appraisal  participants  and 
coming  to  consensus  on  the  draft  changes  that  will  be  incorporated  into 
the  final  findings  briefing.  The  final  findings  briefing  is  updated  to 
reflect  the  accepted  changes.  Also,  at  this  time  any  final  adjustments  to 
the  rating  profile  are  made,  if  appropriate.  The  team  provides  feedback 
on  the  dry-run  to  the  appraisal  team  leader,  who  will  be  presenting  the 
findings,  to  help  him/her  refine  the  delivery  of  the  findings. 


Table  2-33  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team 

Discusses  the  proposed  changes  and  comes  to 
consensus  on  Aem;  provides  feedback  to  the  ATL 
on  the  delivery  of  the  findings. 

Appraisal  team 
leader 

Accepts  feedback  on  the  presentation  of  the 
findings  and  rehearses  rough  points,  where 
necessary;  ensures  that  consensus  on  the  proposed 
changes  is  achieved. 

Session  recorder 

Incorporates  the  accepted  changes  into  the  final 
findings  briefing. 

Appraisal  support 
personnel 

Prepare  slides  and  enough  copies  of  the  final 
briefing  for  the  sponsor  and  the  appraisal  team, 
plus  a  few  extra  in  case  the  sponsor  wants  extras 
for  his/her  own  distribution. 

Table  2-33.  Participants  for  Adjust  Draft  Findings. 
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2.2.13  Adjust  Draft  Findings,  Continued 


Typical  duration  Two  to  four  hours 


Steps 


Table  2-34  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Review  draft 
changes 

Obtain  consensus  on  the  proposed  changes 
provided  by  the  participants.  Not  all  changes 
must  be  accepted;  particularly,  ones  that  do  not 
add  to  the  global  issues  of  the  appraisal  may  not 
be  accepted.  The  team  must  agree  on  the 
changes  before  they  are  made. 

Determine  impact  on 
rating  profile 

Based  on  what  was  heard  in  the  findings 
review,  quickly  review  the  rating  profile  to 
ensure  that  it  is  still  valid  (99%  of  the  time  it 
will  be  fine,  but  you  want  to  make  sure 
findings  stay  consistent  with  the  profile).  If 
necessary,  come  to  consensus  on  rating  profile 
changes. 

Discuss  delivery  of 
findings 

Team  members  may  provide  hints  on  the 
delivery  of  the  findings  to  the  ATL  for 
consideration  in  making  the  message  come 
across  more  clearly  or  simply.  The  ATL  may 
rehearse  alternate  approaches  to  delivering 
findings  that  prove  sensitive  or  more  difficult  to 
articulate. 

Table  2-34.  Steps  for  Adjust  Draft  Findings. 
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2.2.13  Adjust  Draft  Findings,  Continued 


Steps,  continued 


Tailorable 

parameters 


Exit  criteria 


Notes 


Step 

Guidance 

Produce  final 
findings  briefing 

Red-line  changes  are  made  to  the  original  if  red- 
lines  are  being  presented,  or  the  electronic  file 
with  the  briefing  is  updated.  At  this  point, 
ensure  that  all  the  other  pieces  of  the  briefing 
(the  introduction,  rating  profile,  and  next  steps) 
are  incorporated  into  the  final  copy.  Appraisal 
support  personnel  make  the  required  number  of 
slides  and  hard  copies.  Unless  otherwise 
established  ahead  of  time,  only  copies  for  the 
sponsor  and  appraisal  team  are  made. 
Frequently,  sponsors  wish  to  control 
distribution  of  the  findings  briefing.  To 
provide  a  copy  to  those  who  want  one,  while 
still  retaining  a  measure  of  control  over 
distribution,  we  suggest  having  a  sign-in  sheet 
at  the  findings  meeting  with  a  place  to  check  if 
people  want  a  copy  of  the  briefing. 

Table  2-34.  Steps  for  Adjust  Draft  Findings,  continued 


•  The  findings  briefing  may  either  be  revised  electronically  so  a  "fresh" 
copy  is  presented,  or  a  red-lined  copy  of  the  draft  briefing  may  be 
used,  depending  on  the  culture  and  support  resources  available. 
(Doing  a  fresh  copy  implies  in-team  computing  support  which  may  or 
may  not  be  available.) 


•  Ratings  profile  is  finalized. 

•  Final  findings  briefing  is  prepared  for  delivery  to  participants. 


Be  sure  the  ATL  is  comfortable  delivering  the  briefing.  The  two 
rehearsals  provided  by  the  draft  findings  briefing  are  usually  sufficient 
to  reduce  the  nervousness  of  presenting  to  the  sponsor. 
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2.2.14  Present  Final  Briefing 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.14  Present  Final  Briefing  (PB) 


The  purpose  of  Present  Final  Briefing  is  to  provide  the  sponsor  with  the 
agreed-upon  data  from  the  appraisal  and  determine  next  steps  for  use  of 
the  findings  (e.g.,  catalyst  for  process  improvement  effort  or  selection 
of  supplier). 


Present  Final  Briefing  involves  presenting  the  results  of  the  appraisal  to 
the  sponsor  and  usually  to  the  oAer  appraisal  participants  via  a  briefing 
that  synthesizes  the  data  in  a  non-attributable  form  and  provides 
prioritized  findings. 


Table  2-35  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team 
leader 

Presents  the  findings  and  rating  profile  to  the 
sponsor.  Facilitates  any  discussion  after  the 
briefing. 

Appraisal  team 

Notes  feedback  from  sponsor. 

Sponsor 

Accepts  findings  of  appraisal  from  team; 
prioritizes  actions  to  be  taken  based  on 
appraisal  results. 

Appraisal 

participants 

Listen  to  appraisal  results. 

Table  2-35.  Participants  for  Present  Final  Briefing. 


One  to  two  hours 


continued  on  next  page 
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2.2.14  Present  Final  Briefing,  continued 


Steps 


Tailorable 

parameters 


Exit  criteria 


Notes 


Table  2-36  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Coach  sponsor  on 
his/her  expected 
participation  in 
meeting 

Remind  sponsor  that  his/her  reaction  to  the 
findings  will  have  an  impact  on  the  enthusiasm 
with  which  follow  on  actions  are  greeted. 

Remind  all  of 
purpose  of  meeting 

Remind  participants  that  they  have  already  had 
opportunities  for  feedback  -  the  gist  of  this 
session  is  to  summarize  what  has  been  found 
during  the  week. 

Thank  sponsor  and 
participants  for 
cooperation  in  the 
appraisal 

Acknowledge  the  support  personnel  and  all 
those  who  helped  the  appraisal  succeed. 

Present  final  briefing 

The  presenter  goes  through  the  entire  briefing, 
including  process  capability  profile  and  next 
steps;  presenter  asks  for  questions  to  be  held 
until  the  end. 

Open  discussion,  if 
appropriate 

Usually  at  the  end  of  the  briefing,  the  sponsor 
does  a  "thank  you"  message,  provides  initial 
reaction,  and  then  opens  with  his/her  own 
questions  or  opens  the  discussion  to  the  floor. 

Table  2-36.  Steps  for  Present  Final  Briefing. 


•  Number  and  type  of  participants  in  audience. 

•  Level  of  detail  of  findings. 

•  Presenter:  in  some  cases,  having  the  facilitator  present  the  process  of 
coming  to  the  findings  and  the  team  leader  present  the  content  of  the 
findings  may  work  best  (see  tailorable  parameters  in  Present  Draft 
Findings). 


♦  Final  briefing  presentation  completed. 

•  Participant  questions  answered,  or  recorded  as  actions. 


See  Appendix  B  for  rating  profile  example 
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2.2.15  Brief  Sponsor  (optional) 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  Duration 


2.2.15  Brief  Sponsor  (BS) 


The  purpose  of  Brief  Sponsor  is  to  provide  the  sponsor  with  an 
opportunity  to  ask  questions  privately,  obtain  or  provide  feedback  on 
the  appraisal  process,  and  discuss  next  steps  in  more  detail. 


Brief  Sponsor  involves  the  appraisal  team  members  and  sponsors 
having  an  open  discussion  on  the  results  of  the  appraisal,  the  appraisal 
process,  and/or  the  next  steps,  as  appropriate.  No  confidentiality  rules 
are  abrogated  in  this  meeting. 


Table  2-37  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Asks  sponsor  if  any  clarification  or  other 
information  is  needed,  suggests  follow-on 
assignments. 

Appraisal  team 

Answer  sponsor's  questions,  as  appropriate. 

Sponsor 

Asks  any  questions  not  appropriate  for  a  general 
audience;  makes  follow-up  assignments. 

Table  2-37.  Participants  for  Brief  Sponsor. 


Thirty  minutes  to  one  hour 


continued  on  next  page 


2-60 


SECMM-94-06ICMU/SE1-96-HB-004  v1 .1 


2.2.15  Brief  Sponsor  (optional),  continued 


Steps 


Tailorable 

parameters 

Exit  criteria 

Notes 


Table  2-38  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Ask  sponsor  for 
feedback  on  results 

Often  the  sponsor  has  questions  of  clarification 
that  he/she  prefers  not  to  ask  in  a  large  group; 
even  if  there  are  no  specific  questions,  this  is  a 
good  time  to  gauge  the  sponsor's  reaction  - 
sometimes  the  results  are  right  in  line  with  the 
sponsor's  prior  thinking,  sometimes  they  are  a 
surprise. 

Ask  sponsor  for 
feedback  on  process 

Any  feedback  from  the  sponsor  on  the  appraisal 
process  should  be  included  in  the  lessons 
learned  that  are  returned  to  the  SE-CMM 
maintenance  site. 

Discuss  next  steps 
and  add  detail 

At  this  point  the  assignments  for  follow-on 
work  should  be  finalized.  The  facilitator  may 
provide  some  advice  on  how  long  the  follow¬ 
up  activities  will  be  expected  to  take  and  the 
level  of  commitment  required  to  finish  the 
report  and  start  to  develop  an  improvement 
plan. 

Table  2-38.  Steps  for  Brief  Sponsor. 


•  This  is  an  optional  process  element,  so  it  may  be  tailored  out  if 
appropriate. 


•  Sponsor  dismisses  team. 


This  is  an  optional  step,  but  is  frequently  used  as  a  way  to  ensure 
sponsor  follow-through  and  assignment  of  actions.  It  also  provides  the 
sponsor  with  an  opportunity  to  get  a  better  understanding  of  the  findings 
which  he/she  may  not  have  been  comfortable  discussing  in  the  large 
meeting. 
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2.2.16  Conduct  Wrap-Up 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.16  Conduct  Wrap-Up  (CW) 


The  purposes  of  Conduct  Wrap-Up  are  to  obtain  feedback  from  the 
appraisd  team  on  the  appraisal  process  itself,  provide  an  opportunity  for 
consulting  with  the  facilitator  on  moving  forward  with  the  results,  and 
ensure  that  appraisal  materials  are  properly  accounted  for. 


Conduct  Wrap-Up  involves  obtaining  information  about  what  worked 
and  what  did  not  work  from  the  appraisal  team  members  for  feeding 
back  to  the  SAM  maintainers.  It  also  involves  discussing  and  resolving 
findings  and  assignments  from  the  recommendation  report. 


Table  2-39  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Facilitates  gathering  data  on  what  worked  and  did 
not  work  during  the  appraisal. 

Appraisal  team 
members 

Provide  input  into  what  worked  and  what  didn’t 
work. 

Table  2-39.  Participants  for  Conduct  Wrap-Up. 


One  hour 


continued  on  next  page 
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2.2.16  Conduct  Wrap-Up,  Continued 


Steps 


Tailorable 

parameters 

Exit  criteria 

Notes 


Table  2-40  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Conduct  appraisal 
data  gathering 
session 

Getting  the  team  to  write  "what  worked"  and 
"what  didn't  work"  post-it  notes  provides  an 
easy  way  to  gather  data  and  provide  an 
affinitized  list  of  improvement  suggestions  to 
the  maintenance  site  —  what  did  work  is  just  as 
important,  so  that  successful  elements  of  SAM 
don't  get  lost  in  future  revisions 

Verify  findings  and 
recommendations 

If  there  were  areas  of  disagreement  that  came 
out  as  a  results  of  the  final  briefing,  this  time 
can  be  used  to  resolve  how  the  finding  will  be 
presented  in  the  appraisal  report 

Report  assignments 
detailed 

All  appraisal  team  members  should  understand 
their  commitments  to  the  findings  8 
recommendations  report  and  their  deadlines. 

Answer  last-minute 
questions 

Often  appraisal  team  members  have  questions 
about  improvement  plans,  etc.,  that  the 
facilitator  can  spend  some  time  answering 
and/or  providing  references  for. 

Table  2-40.  Steps  for  Conduct  Wrap-Up. 


•  Amount  of  time  spent  in  discussing  lessons  learned. 


•  Appraisal  lessons  learned  are  recorded. 

•  Assignments  are  verified. 


Depending  on  the  team,  there  may  be  resistance  to  the  wrap-up  session 
(everybody  wants  to  go  home  and  get  some  sleep! ! !  ...  or  catch  a 
plane,  etc.).  The  facilitator  should  judge  how  in-depth  to  make  this 
session  based  on  the  state  of  the  team.  A  celebration  of  some  sort  is  not 
out  of  place  if  the  team  members  are  available.  (Often,  however,  the 
facilitators  have  planes  to  catch.) 
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2.3  Post-Appraisal 


Element  title/tag 
Purpose 

Diagram 


2.3  Post-Appraisal  (PO) 


The  purpose  of  the  Post- Appraisal  phase  is  to  finish  preparing  the 
appraisal  reports,  document  lessons  learned  in  the  appraisal,  and 
provide  recommendations  for  action  planning.  Figure  2-4  shows  the 
steps  in  the  Post-Appraisal  phase. 


Figure  2-4.  Diagram  of  Post-Appraisal  Phase. 


continued  on  next  page 
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2.3  Post-Appraisal,  Continued 


Summary 

description 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 

Notes 


Table  2-41  shows  the  steps  for  this  process  element. 


ID 

Activity 

Description 

2.3.1 

Report  lessons 
learned 

Record  lessons  learned,  risks  avoided 
and/or  encountered,  and  suggestions  for 
improving  the  method  to  the  SE-CMM 
maintenance  site. 

2.3.2 

Report  appraisal 
outputs  to  other 
parties 

Identify  appraisal  data  in  a  way  that 
maintains  confidentiality  (i.e.,  "sanitize" 
the  data)  and  provide  to  organization(s) 
gathering  state-of-the-practice  data,  if 
agreed  upon  in  the  preparation  phase. 

2.3.3 

Manage  records 

Properly  dispose  of  intermediate  and 
final  data  generated  by  the  appraisal. 

2.3.4 

Develop  findings- 

recommendation 

report 

Expand  findings  briefing  into  a  report; 
add  recommendations. 

Table  2-41.  Summary  Description  of  Post-Appraisal  Phase. 


Two  to  four  weeks 


•  Reporting  of  appraisal  data  to  other  parties  (e.g.,  SE-CMM 
maintenance  site)  must  be  negotiated  and  agreed  upon  in  the 
preparation  phase. 


•  Lessons  learned  reported  back  to  SE-CMM  maintenance  site. 

•  Appraisal  report  completed  and  distributed  appropriately. 

•  Records  disposed  of  in  accordance  with  the  appraisal  plan. 


One  of  the  typical  traps  in  getting  an  improvement  effort  started  is 
allowing  the  release  of  the  appraisal  report  to  drag  out. 
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2.3.1  Report 

Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 

Notes 


Lessons  Learned 


2.3.1  Report  Lessons  Learned  (LL) 


The  purpose  of  Report  Lessons  Learned  is  to  provide  SAM  maintainers 
with  feedback  on  the  strengths  and  weaknesses  in  the  description  of  the 
method  and  model,  and  provide  opportunities  for  improving  the  method 
and  model. 


Report  Lessons  Learned  involves  describing  the  appraisal  goals  and 
tailoring  decisions  made  by  the  team,  and  recording  any  effects  (positive 
or  negative)  of  either  the  described  or  implemented  method. 


Table  2-42  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Synthesizes  and  records  lessons  learned  from 
appraisal  participation. 

SAM  maintenance 
site 

Accepts  and  addresses  comments  from 
appraisal  team. 

Table  2-42.  Participants  for  Report  Lessons  Learned. 


Two  hours  (to  write  and  send  lessons  learned  information) 


•  Level  of  detail  of  recommendations. 


•  Lessons  learned  sent  to  SAM  maintenance  site. 


Feedback  received  from  appraisal  teams  will  be  a  primary  input  to 
revisions  and  enhancements  to  SAM. 

Feedback  may  be  as  simple  as  a  set  of  post-it  notes,  or  as  elaborate  as  a 
formal  white  paper. 
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2.3.2  Develop  and  Present  Action  Plans 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 

Tailorable 

parameters 

Notes 


2,3.2  Develop  and  Present  Action  Plans  (AP) 


The  purpose  of  action  planning  is  to  provide  management  and  the 
appraisal  sponsor  with  feasible,  prioritized  activities,  costs,  and 
schedules  descriptions  that  are  likely  to  be  implemented;  and  to  deliver 
intended  improvements  from  the  perspectives  of  the  organization's 
technical  and  managerial  levels  . 

Action  planning  requires  willing  and  collaborative  representatives  from 
affected  groups  in  the  organization  to  develop  realistic  task  descriptions, 
resource  cost  estimates,  schedules,  and  perform  risk  assessments  before 
submitting  the  plans  for  approval  and  implementation  by  the  sponsor  or 
management.  The  plans  need  not  be  long  or  detailed,  but  must  motivate 
the  people  who  must  lead,  do,  and  support  the  plans  to  implement  them 
and  provide  sufficient  guidance  to  do  so. 


Table  2-43  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  the  process. 


Role 

Summary 

Action  planners 

Develop  action  plans  to  implement  prioritized 
recommendations  and  prioritize  the  actions  to 
maximize  impact  and  buy-in  from  the 
organization. 

Table  2-43.  Participants  for  Report  Appraisal  Output  to  Other 

Parties. 


•Two  days  to  four  weeks. 


•  Level  of  detail  provided.  At  the  low  end,  only  title,  objective 
(observable  or  measurable),  linkage  to  finding/recommendation,  task 
list,  resources  to  accomplish,  PERT,  and  force  field  analysis 
(pros/cons  risk  assessment)  are  required.  At  the  top  end,  this  could  be 
a  significant  portion  of  an  organization's  annual  tactical  or  operational 
planning. 


The  levels  of  detail,  resources,  and  schedule  to  do  this  planning  are  to 
be  negotiated  during  the  preparation  phase  with  the  sponsor. 
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2.3.3  Perform  Actions 


Element  title/tag 

Purpose 

Summary 

description 

Major 

participants 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 

Notes 


2.3.3  Perform  Actions  (PA) 


The  purpose  of  Perform  Actions  is  to  implement  or  deploy  the  approved 
action  plans. 

Perform  Actions  involves  the  sponsor  and  affected  groups  (including 
management)  reviewing  the  action  plans,  and  identifying  resources  and 
appropriate  start  dates  for  those  efforts  approved.  Then  deploying  the 
resources  according  to  the  plan(s)  and  tracking  and  overseeing  them  like 
any  other  project  (see  PAl  1 :  Monitor  and  Control  Technical  Effort). 


Table  2-44  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Sponsor 

Accepts,  prioritizes,  and  provides  resources 
and  leadership  to  implement  the  plan(s). 

Action  Plan  team(s) 

In  accordance  with  project  PAs  herein,  perform 
action  plans  to  achieve  expected  results. 

Table  2-44.  Participants  for  Manage  Records. 


•  From  a  few  weeks  to  an  ongoing  effort  each  (from  3  to  5  distinct 
improvement  action  plans  are  typical  outputs  of  an  appraisal) 


•  Many  -  these  will  be  determined  by  how  the  organization  uses  teams, 
task  forces,  inter-disciplinary  groups,  etc.  to  deploy  changes. 


•  Plan  exit  criteria  are  met  (success  or  failure  is  achieved). 


A  good  resource  for  how  to  deploy  change  is  Peter  Scholtes'  "Team 
Handbook." 


2-68 


SECMM-94-06ICMU/SEI-96-HB-004  v1 .1 


2.3.4  Develop  Findings  and  Recommendations  Report 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 

Notes 


2.3.4  Complete  and  deliver  findings  and  recommendations  report  (FR) 


The  purpose  of  developing  the  F&R  report  is  to  provide  the  organization 
with  documentation  of  root  causes  of  problems  that  are  widely  accepted 
in  the  organization,  and  recommendations  for  actions  to  remove  them. 


The  F&R  report  copies  and  provides  details  or  explanation  of  intent 
beyond  the  findings  slide  and  process  area  profile.  For  each  finding, 
the  report  is  to  contain  a  recommendation  that  addresses  that  finding. 
These  recommendations  should  focus  the  organization  on  increasing 
throughput  and  decreasing  work-in-progress  (WIP)  in  SE  processes 
(process  areas  as  deployed  in  the  organization). 


Table  2-45  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Leader 

Collects  and  leads  synthesis  of 
recommendations  from  findings. 

Appraisal  team 

Perform  writing  assignments  and  review  and 
provide  improvement  recommendations  to  other 
authors'  work. 

Facilitator 

Acts  as  a  reviewer  of  the  report. 

Table  2-45.  Participants  for  Report  Lessons  Learned. 


•  Four  work  days  over  one  to  three  weeks 


•  Level  of  detail  of  document  and  of  explanatory  material  supporting 
findings  and  recommendations. 


•  Report  delivered  and  accepted  or  approved  by  sponsor. 


We  advocate  in  the  F&R  template  and  SAM  training  pages  that 
Goldratt's  “Theory  of  Constraints”  be  the  basis  of  improvement 
recommendations.  It  focuses  recommendation  developers  on  throughput 
and  the  elimination  sub-optimization  conditions  that  cause  work-in¬ 
progress  to  pile  up  before  process  bottlenecks.  A  worksheet  including 
some  techniques  that  might  be  helpful  is  included  in  Appendix  E. 
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Chapter  3: 
Processes 


Purpose 


Introduction 


Facilities  support 


Additional 

appraisal 

guidance 


Guidance  on  Important  SAM 
for  Initiating  an  Appraisal 


The  purpose  of  this  chapter  is  to  provide  readers  who  are  contemplating 
initiating  a  SAM  with  information  that  will  help  them  properly  resource 
and  prepare  for  the  appraisal. 


The  preparation  phase  is  typically  initiated  six  to  eight  weeks  prior  to  the 
on-site  week.  The  primary  activities  performed  during  the  preparation 
phase  are 

•  Establish  appraisal  objectives. 

•  Determine  boundaries  to  be  placed  on  the  organization  for  the  purpose 
of  establishing  the  scope  of  the  entity  to  be  appraised. 

•  Select  projects,  or  major  aspects  of  a  project,  that  will  receive  the 
questionnaires. 

•  Select  appraisal  team  and  site  coordinator. 

•  Select  systems  engineering  leads,  support  functions,  and  practitioner 
groups. 

•  Obtain  facilities  zind  support  material. 

•  Coordinate  schedules,  particularly  senior  management. 

•  Administer  questionnaire. 

A  checklist  to  assist  the  site  coordinator  is  provided  in  Appendix  F. 
Many  of  these  issues  require  guidance  to  optimize  the  organization's  use 
of  SAM.  This  chapter  provides  some  of  that  guidance. 


The  on-site  appraisal  phase  is  very  intensive  for  the  appraisal  team.  To 
maximize  their  efficiency,  it  is  recommended  that  a  large  conference 
room  which  can  be  secured  be  made  available  for  the  on-site  effort. 
Other  critical  support  facilities  include  computers,  printers,  display 
capability,  and  reproduction  capability.  Recommended  support  facilities 
include  hotel  rooms,  food,  and  secretarial  support. 


All  three  appraisal  phases  are  discussed  in  detail  in  Chapter  2,  "Process 
Element  Summaries."  In  each  process  element,  guidance  is  provided  to 
help  appraisers  avoid  typical  pitfalls  associated  with  that  process 
element. 


continued  on  next  page 
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Chapter  3:  Guidance  on  Important  SAM 
Processes  for  Initiating  an  Appraisal,  continued 


In  this  chapter  The  following  table  provides  a  guide  to  the  information  found  in  this 
chapter. 


Topic 

See  Page 

3.1  Using  Business  Goals  in  SAM 

3-3 

3.2  Tailoring  SAM  Based  on  Organizational  Context 

3-4 

3.3  Selecting  Appraisal  Personnel 

3-7 

3.4  Selecting  Projects  to  be  Appraised 

3-9 

3.5  Using  the  Appraisal  Questionnaire 

3-11 

3.6  Developing  the  Rating  Profile 

3-15 

3.7  Developing  Findings 

3-25 

3-2 
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3.1  Using  Business  Goals  in  SAM 


Assumptions  The  following  discussion  presumes  that  an  organization  using  SAM  has 
a  strong  set  of  deployed  business  goals  at  its  disposal.  The  creation  and 
deployment  of  business  goals  is  a  task  of  significant  undertaking  and  is 
well  beyond  the  scope  of  this  method  description. 


An  organization  that  has  a  strong  sense  of  its  mission  and  business 
goals  can  optimize  the  use  of  SAM  by  analyzing  the  model  in  relation  to 
those  goals  and  deciding  on  a  focus  for  the  appraisal  that  may  be 
narrower  than  the  entire  model,  but  which  can  provide  more  direct  and 
focused  information  on  the  process  improvement  needs  of  the 
organization. 

Alternatively,  a  broad  focus  may  still  be  desired  for  data  gathering; 
however,  the  focus  of  findings  and  recommendations  can  be  limited  to 
those  which  directly  impact  the  business  goals.  This  provides  the 
breadth  of  data  gathering  that  allows  management  a  benchmark  of 
overall  capability  while  still  ensuring  a  focus  on  improvement  that  is 
directly  tied  to  the  business  needs  of  the  enterprise. 
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Refining 
Appraisal  Focus 


3.2  Tailoring  SAM  Based  on  Organizational  Context 


Many  factors 
influence 
organizational 
processes 


Organization 

structure 


Figure  3-1  illustrates  some  of  the  key  influences  organization’s  must 
face  when  attempting  improvement  using  any  reference  model,  such  as 
the  SE-CMM.  The  discussion  following  Figure  3-1  is  intended  to  assist 
organizations  in  determining  some  of  the  tailoring  of  the  SAM  that  may 
be  necessary  to  support  improvement  in  their  particular  context. 

Organizational  Guidance  by  SE-CMM 


Figure  3-1.  Key  Influences  of  Organizational  Improvement. 


To  a  certain  extent,  SAM  has  been  designed  to  optimize  appraisal 
efficiency  in  an  organizational  setting  that  uses  projects  as  the  main 
meMs  of  accomplishing  work.  These  projects  are  assumed  to  be  led  by 
an  individual  or  team  who  have  resource  planning,  allocation,  and 
monitoring  responsibility,  as  well  as  responsibility  for  the  quality  and 
quantity  of  product  delivered.  Other  management  and  control  functions, 
such  as  staffing,  career  development,  and  administration,  may  or  may 
not  be  performed  within  the  project  boundaries. 

Common  organizational  structures  that  adapt  well  to  SAM  include  a 
project  structure  where  all  management,  staffing,  and  administrative 
functions  come  within  the  purview  of  the  project;  a  matrix  structure, 
where  the  project  planning,  resource,  monitoring  and  performance  are 
the  responsibility  of  the  project  with  staffing  and  other  administration 
being  handled  by  a  "functional"  or  "central"  organization;  and  integrated 
product  development  teams,  where  teams  are  created  from  functional  or 
product  line  groups  which  then  have  the  same  general  responsibilities  as 
a  project,  but  typically  for  a  smaller  scope  for  each  team. 

In  some  environments,  a  functional  structure,  where  the  resource 
planning  and  control  for  product  development  are  under  the  management 
of  a  central  or  functional  organization,  can  be  a  more  challenging 
environment  in  which  to  use  SAM  to  obtain  a  snapshot  of  organization- 
wide  systems  engineering  capability. 


continued  on  next  page 
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3.2  Tailoring  SAM  based  on  Organizational  Context,  continued 


Using  SAM  in 
functional 
organization 
structure 


SAM  assumes  that  there  is  someone,  in  SAM  called  the  Systems 
Engineering  Leader,  who  is  a  single  point  of  information  for  a  broad 
base  of  topics  related  to  the  systems  engineering  functions  being 
performed  in  projects  within  an  organization.  If  three  or  four  of  these 
SE  leads  are  interviewed  using  SE-CMM  as  the  reference  model,  the 
gaps  in  information  are  typically  relatively  small,  and  easily  filled  via  the 
practitioner  group  discussions. 

In  some  organizations  the  resource  control  is  held  by  a  functional 
manager,  either  related  to  life  cycle,  e.g.,  a  requirements  manager, 
design  manager,  integration  manager,  etc.;  or  by  product  line,  e.g.,  the 
product  line  A  manager,  product  line  B  manager,  etc.  The  "projects" 
constituted  to  support  this  type  of  organization's  needs  may  be  small 
enough  that  the  ongoing  overhead  of  a  project  management  structure  is 
inefficient,  or  this  structure  may  support  business  goals  of  the 
organization  in  some  way  other  structures  cannot. 

The  challenge  for  applying  SAM  is  in  trying  to  find  a  single  interview 
source  for  the  broad  information  gathering  needed  at  the  beginning  of 
SAM.  This  may  not  be  possible.  Where  infeasible,  tailoring  the 
method  to  achieve  the  goals  of  getting  both  breadth  of  information 
typically  provided  by  the  SE  leads,  and  depth  of  information  typically 
provided  by  the  practitioner  groups,  is  the  likely  alternative.  This  may 
entail  changing  the  on-site  schedule  for  the  appraisal,  re-defining  the 
"organization"  being  appraised,  distributing  the  questionnaire  across  a 
broader  range  of  projects  and  then  selecting  interviewees  based  on  the 
breadth  of  knowledge  exhibited  by  their  questionnaire  responses,  or 
other  alternatives. 


continued  on  next  page 
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3.2  Tailoring  SAM  based  on  Organizational  Context,  continued 


Using 

organizational 

development 

resources 


Organizational  development(OD)  is  a  discipline  within  the  social  and 
decision  sciences  which  attempts  to  understand  the  behavior  of  large  and 
small  workgroups,  as  well  as  design  and  execute  interventions  needed 
to  optimize  organizational  functioning,  enable  needed  changes  to  occur, 
and/or  positively  influence  aberrant  organizational  behaviors.  Many 
product  development  enterprises  have  OD  staff,  either  in  the  training 
department,  or  in  a  specific  organization  called  Organizational 
Development  or  something  similar.  OD  specialists  are  frequently  a 
valuable  source  in  determining  how  to  tailor  SAM  for  a  particular 
organizational  context,  especially  one  that  does  not  immediately  appear 
to  follow  one  of  the  patterns  expected  by  SAM.  They  can  also  be 
helpful  in  selecting  appraisal  team  members,  appraisal  participants, 
projects,  and  improvement  strategies,  as  well  as  help  to  plan  and 
implement  the  organizational  improvement  plan. 

OD  staff  may  be  particularly  helpful  in  determining  the  most  appropriate 
project  lead/functional  management  mix  for  functional  style 
organizations. 


Product  line  Many  organizations  today  are  using  product  lines  as  a  method  of 

focus  defining  organizational  boundaries.  When  applying  SAM  in  such  an 

organization,  an  appraisal  focusing  within  a  single  product  line  is  likely 
to  produce  the  most  actionable  results.  Crossing  product  lines  can  give 
a  perspective  on  the  overall  process  strengths  and  weaknesses  of  the 
organization,  but  is  less  likely  to  come  up  with  focused  findings  that  can 
be  leveraged  from  for  significant,  timely  improvement. 
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3.3  Selecting  Appraisal  Personnel 


Guidance  for 
selecting  the 
appraisal  team 


•  Be  advocates  of  process  improvement. 

•  Be  credible  with  both  management  and  participants. 

•  Be  involved  in  action  planning  and  the  subsequent  improvement 
effort. 

•  Have  good  communication  skills. 

•  Have  a  positive  and  encouraging  attitude. 

If  the  organization  has  any  previous  experience  with  organizational 
appraisals  and  ensuing  process  improvement  activities,  such  as 
improvement  based  on  the  CMM  for  Software,  it  can  be  advantageous 
to  include  a  person  who  has  been  involved  in  those  activities.  Such  a 
person  brings  appraisal  experience  from  within  the  organizational 
culture,  and  can  be  used  as  a  source  to  gain  leverage  from  process 
improvement  activities  that  have  worked  within  the  organizational 
culture.  These  individuals  are  also  likely  candidates  to  be  SAM 
facilitators. 

continued  on  next  page 


The  facilitators  and  appraisal  team  members  drawn  from  the 
organization  work  as  a  single  team  during  the  entire  on-site  phase.  The 
appraisal  team  will  analyze  data,  perform  all  of  the  interviews,  and 
develop  the  findings.  They  are  also  responsible  for  developing  the 
recommendations  to  the  findings  which  are  published  in  the  appraisal 
report,  and  should  be  involved  in  carrying  process  improvement 
activities  throughout  the  organization.  To  assure  transition  from  the 
appraisal  to  a  robust  improvement  effort,  the  appraisal  team  members 
should 
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3.3  Selecting  Appraisal  Personnel,  continued 


Why  use  systems 
engineering  leads 
and  practitioners 
as  primary  data 
sources? 


The  SE-CMM  appraisal  method  is  structured  to  examine  two  views  of 
the  entity  being  appraised,  typically  project  and  organizational  support. 
The  objective  is  to  get  as  broad  a  view  of  the  organization  as  possible 
while  still  maintaining  control  of  the  data  gathering  process.  The  project 
lead  systems  engineers  are  typically  the  only  ones  who  complete  the 
questionnaire  since  they  have  a  broader  view  of  the  tasks  being 
performed  in  systems  engineering  and  associated  support  functions  than 
most  other  roles  in  a  product  development.  The  questionnaires  are 
essential  to  establishing  the  basis  for  data  gathering  for  the  entire  on-site 
week.  The  practitioners  expand  the  appraisers'  view  of  the 
organization,  providing  information  on  what  is  being  done  on  other 
projects  or  areas  of  the  organization  besides  those  targeted  for  in-depth 
data  gathering.  The  appraisal  team  may  choose  to  have  select  people  in 
the  practitioner  group  complete  the  questionnaire,  but  this  is  typically 
kept  to  a  limited  number  for  the  purpose  of  data  and  resource 
management. 


On  some  appraisals  the  appraised  organization  may  be  a  large  project. 
In  that  situation  the  "projects"  may  be  large  subsystems  or  segments  of 
the  project.  The  project  lead  systems  engineers  would  then  be  the 
segment  lead  systems  engineers,  and  the  practitioners  would  be  drawn 
from  other  project  segments. 
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3.4  Selecting  Projects  to  Be  Appraised 


Project  selection 
considerations 


Project  location 


The  types  of  projects  selected  to  participate  in  the  appraisal  are  based  on 
the  goals  of  the  appraisal.  Table  3-1  lists  some  considerations  in 
selecting  projects. 


Goal  of  the  Appraisal 

Type  of  Projects  to  Select 

Understand  domain-related 
issues. 

Select  projects  within  desired  domain. 

Note  that  the  domain  can  be  focused  on 
any  one  of  the  following  factors:  industry, 
technology  base,  customer  type,  project 
complexity,  etc. 

Understand  deployment  of 
new  organization^ 
practices. 

Select  new  projects  that  have  started  since 
the  deployment  of  new  practices. 

Determine  overall  capability 
of  the  organization. 

Select  projects  that  are  expected  to  be 
representative  of  the  organizational 
capability. 

Determine  progress  of 
process  improvement 
activities. 

Select  projects  that  have  been  the  pilots 
for  process  improvements. 

Table  3-1.  Project  Selection  Considerations. 


Although  it  is  not  required  by  SAM  that  the  projects  of  an  appraisal  be 
near  each  other  geographically,  the  logistics  considerations  associated 
with  a  multi-site  appraisal  should  be  carefully  considered.  The  appraisal 
interview  and  feedback  process  assumes  that  the  interviewers  will  be 
available  for  multiple  sessions  during  the  on-site  week.  Tailoring  to 
accommodate  multi-site  appraisals  should  account  for  the  need  for  the 
interview/feedback  loop  if  activities  are  considered  for  resequencing. 


continued  on  next  page 
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3.4  Selecting  Projects  to  be  Appraised,  continued 


Guidelines  for  There  are  two  issues  associated  with  the  selection  of  the  practitioner 

selecting  groups: 

practitioner 

groups  •  Ensuring  appropriate  functional  areas  are  represented. 

•  Ensuring  the  right  type  of  person  is  participating  in  the  group. 

The  practitioner  groups  should  represent  the  primary  systems 
engineering-related  tasks  performed  within  the  organization  (e.g., 
analysis,  requirements,  test).  However,  it  is  important  to  include  the 
support  organizations  (specifically  quality,  configuration  management, 
and  training)  even  if  these  functions  are  not  encompassed  structure- 
wise,  in  the  organization  being  appraised.  It  is  also  recommended  that 
practitioners  from  specialty  disciplines  who  support  systems 
engineering  (e.g.,  human  factors,  reliability,  manufacturing)  be  included 
in  the  practitioner  sample. 

The  people  who  are  selected  to  be  in  the  practitioner  groups  should 

•  Be  opinion  leaders  who  are  credible  with  their  peer  group. 

•  Be  drawn  from  areas  widespread  throughout  the  organization. 

•  Be  willing  to  communicate  and  express  their  opinions. 

•  Be  able  to  talk  freely  and  supply  useful  information. 

•  Not  suppress  or  intimidate  any  conversation. 

To  ensure  candid  conversation  in  the  practitioner  groups,  it  is  typically 
not  recommended  to  have  line  management  represented.  If  one  of  the 
goals  of  the  appraisal  is  to  assess  management  issues,  the 
recommendation  is  to  have  an  entire  practitioner  group  composed  of 
managers. 
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3.5  Using  the  Appraisal  Questionnaire 


Questionnaire 

distribution 


Recommended 

questionnaire 

recipients 


The  questionnaire  is,  at  a  minimum,  distributed  to  the  systems 
engineering  leads  or  equivalent.  However,  depending  upon  the 
responsibilities  and  visibility  of  the  systems  engineering  leads  within  the 
organization,  the  questionnaires  for  some  of  the  process  areas  may  be 
distributed  to  people  who  are  more  familiar  with  the  area  covered  by  the 
process  area  than  the  systems  engineering  lead.  Specifically,  the 
organizational  process  areas  should  be  completed  by  the  appropriate 
subject  matter  experts. 


To  maximize  the  accuracy  of  initial  responses  to  the  questions  in  the  SE- 
CMM  questionnaire,  it  is  recommended  that  the  questionnaires  be 
distributed  to  individuals  with  the  skills  and  roles  expressed  in  Table  3- 
2. 


Process  Area 

Primary  Recipients 

Secondary 

Recipients 

01;  Analyze  Candidate 
Solutions 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Any  senior  practitioner 
with  significant  system 
design  experience 

02:  Derive  and 

Allocate  Requirements 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

03:  Develop  Physical 
Architecture 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

04:  Integrate 
Disciplines 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Senior  specialty 
engineers  (e.g., 
reliability,  safety, 
manufacturing,  human 
factors)  working  on  the 
projects  selected  for 
appraisal 

05:  Integrate  System 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

06:  Understand 
Customer  Needs  and 
Expectations 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

•  Technical  marketing 
personnel 

•  Proposal  personnel 

•  Customer  service 
personnel 

Table  3-2.  Questionnaire  Distribution  Table. 


continued  on  next  page 
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3.5  Using  the  Appraisal  Questionnaire,  Continued 


Recommended 

questionnaire 

recipients, 

continued 


Process  Area 

Primary  Recipients 

Secondary 

Recipients 

07:  Verify  and 

Validate  System 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

System  verification 
manager  or  senior  test 
engineers 

08:  Ensure  Quality 

Senior  project-level 
quality  manager  or  lead 
(in  environments  with 
shared  quality 
leadership 

responsibility,  systems 
engineering  lead  for  the 
project) 

•  Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

•  Organizational 
quality  manager,  total 
quality  management 
coordinator 

09:  Manage 
Configurations 

Senior  project-level 

CM  manager  for  the 
projects  selected  for 
appraisal 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

10:  Monitor  and 
Control  Technical 

Effort 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

1 1 :  Plan  Technical 
Effort 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

12:  Manage  Risk 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Project  or  program 
manager  for  the 
projects  selected  for 
appraisal 

13:  Define 
Organization's 

Systems  Engineering 
Process 

Individuals  responsible 
for  defining 
organization-level 
processes;  may  be  part 
of  the  quality 
leadership  area, 
policies/procedures 
area,  or  other  support 
group 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Table  3-2.  Questionnaire  Distribution  Table,  continued 


continued  on  next  page 
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3.5  Using  the  Appraisal  Questionnaire,  continued 


Recommended 

questionnaire 

recipients, 

continued 


Process  Area 

Primary  Recipients 

Secondary 

Recipients 

14:  Improve 
Organization's 

Systems  Engineering 
Processes 

Individuals  responsible 
for  deploying 
organization-level 
process  improvement 
activities;  may  be  part 
of  the  quality 
leadership  area, 
policies/procedures 
area,  or  other  support 
group 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

15:  Manage  Product 
Line  Evolution 

Individuals  at 
organization  level 
responsible  for 
strategic  product  line 
positioning  and 
advancement;  may  be 
in  R&D,  technical 
marketing,  or  other 
support  structure 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

16:  Manage  Systems 
Engineering  Support 
Environment 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Individuals  at 
organization  level 
involved  in  deploying 
new  development 
technologies 

17:  Manage  Systems 
Engineering  Training 

Individuals  responsible 
for  organization-level 
training  planning, 
development,  and 
deployment;  may  be 
part  of  an  R&D  group, 
training  department,  or 
other  support  structure 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Table  3*2.  Questionnaire  Distribution  Tabie,  continued 
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3.5  Using  the  Appraisal  Questionnaire,  Continued 


Questionnaire 

administration 


The  questionnaire  must  be  completed  prior  to  the  on-site  phase.  We 
recommend  that  it  be  administered  with  a  facilitator  or  site  coordinator 
present  and  administered  to  the  systems  engineering  leads  and  other 
appropriate  personnel  as  a  group,  so  that  common  questions  can  be 
answered.  The  typical  question  regards  organizational  roles.  The 
person  administering  the  questionnaire  should  emphasize  that  the  SE- 
CMM  is  role  independent.  The  questionnaire  only  asks  if  specific  tasks 
are  performed  on  a  project,  not  if  they  are  performed  by  a  specific 
person. 
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3.6  Developing  the  Rating  Profile 


Introduction 


Diagram  of 
rating  profile 
development 
process 


One  of  the  results  of  the  on-site  phase  is  a  rating  profile  covering  the 
appraised  process  areas.  The  rating  profile  correlates  closely  with  the 
appraisal  findings,  and  the  two  are  developed  in  a  closely  coupled 
process. 

The  rating  profile  is  developed  and  refined  at  specific  points  in  the 
appraisal  process,  as  detailed  in  the  process  element  summaries  in 
section  1.4  and  in  Chapter  2.  At  the  end  of  the  primary  data-gathering 
activities,  prior  to  generating  preliminary  findings,  the  team  reviews  its 
data  tracking  sheets  (DTSs)  to  formulate  a  preliminaiy  rating  for  each 
process  area.  The  DTS  is  a  support  tool  introduced  in  Appendix  C  to 
help  organize  the  data  obtained  by  the  appraisal  team.  Ratings  are  based 
on  the  degree  to  which  the  appraised  entity  performs  all  of  the  practices 
at  a  given  level,  in  the  judgment  of  the  appraisal  team.  Issues  raised  in 
generating  preliminary  ratings  can  be  addressed  during  the  preliminary 
feedback  sessions.  The  rating  is  then  refined  into  a  draft  rating  in 
conjunction  with  developing  the  draft  findings.  Prior  to  drafting  final 
findings,  the  team  once  again  reviews  the  DTSs  to  formulate  the  final 
rating  for  each  of  the  process  areas. 
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Figure  3-2.  Rating  Profile  Development  Process. 
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3.6  Developing  the  Rating  Profile,  continued 


Data  tracking 
sheet 


Data  Tracking 
Algorithm 


The  DTS  is  the  primary  data  management  tool  used  by  the  appraisal 
team  during  the  on-site  phase.  The  responses  to  the  questionnaires  are 
transcribed  onto  the  DTS  at  the  beginning  of  the  on-site  phase.  Trends 
or  inconsistencies  in  data  can  be  identified  more  easily  in  the  DTS 
format,  which  forms  the  basis  for  developing  the  exploratory  questions 
asked  of  the  systems  engineering  lead.  Information  obtained  from  the 
systems  engineering  lead  and  practitioner  interviews  that  supports 
(corroborating  data)  or  contradicts  (opposing  data)  the  premise  that  the 
base  or  generic  practice  characteristic  is  exhibited  is  then  recorded  on  the 
DTS.  All  information  recorded  on  the  DTS,  in  addition  to  the  appraisal 
team's  notes  from  interviews  and  reviews  of  supporting  material,  is 
used  to  develop  the  rating  profile. 

The  DTS  format  also  facilitates  identifying  where  no  additional 
supporting  data  have  been  obtained.  This  permits  directing  further 
interviews,  particularly  the  practitioner  interviews,  to  fill  in  any 
information  gaps.  Conscientious  data  management  throughout  the  on¬ 
site  phase  significantly  improves  the  efficiency  of  the  whole  appraisal 
process  and  facilitates  the  rating  development. 


The  data  tracking  algorithm  is  used  iteratively  throughout  the  on-sight 
week  to  assist  the  appraisal  team  in  estimating  the  organizational 
maturity  level  for  each  Process  Area.  The  estimate  is  likely  to  change 
over  the  course  of  the  on-site  week.  The  ratings  are  described  as 
minimum  percentages  for  the  Base  and  Generic  Practices,  as  shown  in 
Table  3-3  below.  These  percentages  refer  to  the  number  of  “yes” 
responses  out  the  total  number  of  “yes”  and  “no”  responses  provided  by 
the  SE  Leads. 

No  ratings  estimates  are  shown  outside  of  the  appraisal  team  until  the 
Final  Briefing  of  the  on-site  week. 

The  initial  calculation  of  the  algorithm,  known  as  the  Questionnaire 
Rating,  yields  a  reading  of  the  organization’s  maturity  rating  based  on 
the  answers  to  the  questionnaire.  The  next  calculation  of  the  algorithm, 
known  as  the  Preliminary  Rating,  is  based  on  aggregated  data  from  the 
questionnaires  plus  the  SE  Lead  and  Practitioner  interviews.  It 
commonly  differs  from  the  Questionnaire  Rating.  The  next  calculation, 
known  as  the  Draft  Rating,  is  based  on  more  data  from  SE  Leads. 
Similarly,  it  commonly  differs  from  the  Preliminary  Rating.  The  final 
calculation  of  the  algorithm  is  based  on  further  data  from  Practitioners 
and  SE  Leads,  and  is  known  as  the  Final  Rating.  The  Final  Rating 
rarely  differs  from  the  Draft  Rating,  and  is  the  only  rating  shown 
outside  the  appraisal  team. 
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3.6  Developing  the  Rating  Profile,  continued 


Considerations 
in  Calculating 
the  Algorithm 


Base 

Practices 

GP  2.XX 

GP  3.XX 

GP  4.XX 

GP  5.XX 

Ratings 

Responses 


In  calculating  the  algorithm,  the  appraisal  team  is  to  consider  the  number 
of  “yes”  responses  out  the  total  number  of  “yes”  and  “no”  responses 
provided  by  the  SE  Leads.  Consider,  for  example,  calculating  the 
Questionnaire  Rating  from  responses  to  the  six  Base  Practices  of  the 
Analyze  Candidate  Solutions  PA.  If  three  SE  Leads  complete  the 
questionnaire,  providing  16  “yes”  answers  out  of  a  total  of  18  possible 
responses,  then  the  algorithm  would  indicate  that  level  1  might  be 
achieved.  Legitimate  “N/A”  responses,  which  would  be  rare,  are  not 
counted  in  the  total  of  possible  responses. 

In  order  for  a  Process  Area  to  be  considered  at  a  particular  level,  the 
percentage  of  “yes”  answers  must  satisfy  the  percentages  shown  in 
Table  3-3.  For  example,  for  the  appraisal  team  to  judge  the  organization 
at  level  2  in  a  particular  Process  Area,  all  (100%)  of  the  Base  Practices 
and  90%  of  the  Level  2  Generic  Practices  must  be  answered  with  a 
“yes.” 

In  calculating  the  Preliminary  Rating,  the  data  from  SE  Lead  and 
Practitioner  interviews  will  often  change  the  count  sof  “yes”  and  “N/A” 
responses.  That  is,  the  Preliminary  Rating  will  often  differ  from  the 
Questionnaire  Rating. 


Level  1 

Level  2 

Level  3 

Level  4 

Level  5 

90% 

100% 

100% 

100% 

100% 

- 

90% 

100% 

100% 

100% 

- 

- 

90% 

100% 

100% 

- 

- 

- 

100% 

100% 

- 

- 

- 

- 

100% 

Table  3-3  Rating  Table 


In  general,  persons  responding  to  the  questionnaire  should  be 
encouraged  to  provide  “yes”  or  “no”  answers.  In  cases  where  this 
proves  to  be  difficult,  respondents  should  include  explanatory 
comments  on  the  questionnaire.  However,  there  remain  cases  where 
“not  applicable”  (N/A)  or  “don’t  know”  are  the  only  appropriate 
answers.  Such  responses  are  opportunities  for  the  appraisal  team  to 
clarify  through  the  use  of  Exploratory  Questions.  Each  of  the  possible 
responses  is  discussed  below. 
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3.6  Developing  the  Rating  Profile,  Continued 


Response 

Comments 

Yes 

Represents  conformity  to  the  Base  or  Generic 
Practice.  Although  the  SE  Lead  generates  the 
response,  the  appraisal  team  must  ultimately 
judge  the  response  in  the  light  of  subsequent 
appraisal  interviews. 

No 

Treat  legitimate  old  project  no’s  as  n/a’s.  For 
example,  an  old  project  that  didn’t  follow  the 
requirements  allocation  process  because  it  did 
not  exist  when  the  project  started  should  not  be 
penialized  with  a  “no.”  Instead  allow  the  SE 
Lead  to  answer  “n/a”  to  such  questions. 

N/A 

The  appraisal  team  (not  the  SE  lead)  must  be 
the  judge  of  whether  an  n/a  is  legitimate. 
Similarly,  the  appraisal  team  must  be  the  judge 
of  when  a  legacy  project  can  legitimately  claim 
its  waiver  of  a  practice. 

Don’t  Know 

“Don ’t  know's"  should  be  examined  by  the 
appraisal  team.  Some  of  them  are  legitimate; 
e.g.,  SE  Lead  may  not  know  about  the 
allocation  of  resources  to  the  organization’s 
training  program.  Or,  if  questionnaire  is  given 
to  a  functional  lead,  like  CM  or  QA,  then  they 
may  not  be  expected  to  know  detailed 
information  about  categories  other  then  their 
own.  These  cases  should  be  treated  like 

N/A's.  But,  in  general,  answers  to  project, 
engineering,  and  most  organizational  questions 
should  be  known  by  the  SE  Leads.  For  these 
cases,  don’t  know's  represent  a  lack  of 
adequate  institutionalization,  and  should  be 
treated  as  no's. 

3-18 


SECMM-94-06ICMU/SEI-96-HB-004  v1 .1 


3.6  Developing  the  Rating  Profile,  continued 


DTS  Figure  The  following  is  a  condensed  version  of  the  data  tracking  sheet  found  in 

Appendix  C.  _ 


Questionnaires 

Initial 

A 

B 

C 

D 

E 

F 

Projected 

Rating 

PA  01 :  Analyze  Candidate  Solutions 

n 

Base  Practices: 

Establish  Evaluation  Critena 

1.2 

Define  Approach 

1.3 

Identify  Additional  Alternatives 

[.r 

Analyze  Candidate  Solutions 

1.5 

Select  Solutions 

1 

Capture  Results 

n 

Generic  Practices: 

Allocate  Resources 

"TTT 

Assign  Responsibilities 

2.1.3 

Document  Approach 

2.1.4 

Provide  Tools 

2.1.5 

Ensure  Training 

rr.^> 

Plan  Performance 

^.2.r 

Follow  Plans 

2.2.1 

Do  Configuration  Management 

2.3.1 

Verify  Process  t:ompliance 

2.3.2 

Audit  Work  Products 

2.4.1 

Track  with  Measurement 

2.4^ 

'I'ake  (L!orrective  Action 

S.i.i 

Standardize  Process 

5.1.2 

Tailor  Standard  Process 

3.2.1 

Follow  Defined  Process 

3.2.2 

Perform  l^eer  Reviews 

5.2.3 

Use  Process  Data 

4.'n" 

Establish  Quality  Goals 

4.2.1 

Determine  Process  Capability 

4.2.2 

Use  Process  Capability 

5.1.1 

Establish  Process  Goals 

5.1.2 

Improve  Defined  Process 

'5. 51" 

Do  Causal  Analysis 

5.2.2 

Eliminate  Defect  Causes 

Figure  3-3.  Condensed  Data  Tracking  Sheet. 
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3.6  Developing  the  Rating  Profile,  Continued 


DTS  Figure, 
continued 


01:  Analyze  Candidate  Solutions 

Base  Practices: 

1. 1  tstabiish  Evaluation  Cnteria 

1 . 2  Define  Approach 

1.3  icientity  Additional  Alternatives 

T  4  Analyze  ^[iandidate  Solutions 

1 .5  Select  Solutions 

16  Capture  Results 

Generic  Practices: 


01 :  Analyze  Candidate  Solutions 

kjase  Practices: 

i .  1  tstabiish  Evaluation  Criteria 

1.2  Define  Approach 

1 .3  identify  Additional  Alternatives 

1 .4  Analyze  Candidate  Soluaons 

T.5  Select  Solutions 

1.6  Capture  kesults  ~ 

aenenc  P^ractjces: 


<same  as  above> 


Findings  Review 


Prelim 

Rating 

rinal 

Jth  Rating 


<same  as  above> 


Figure  3-3.  Condensed  Data  Tracking  Sheet,  continued 

continued  on  next  page 
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3.6  Developing  the  Rating  Profiie,  Continued 


Preliminary 

ratings 


Draft  and  Hnal 
ratings 


Using  rating  data 


No  attempt  should  be  made  to  develop  a  preliminary  rating  until  all  of 
the  primary  data  have  been  collected  (all  initial  systems  engineering 
leads  and  practitioners  have  been  interviewed,  and  supporting  material 
has  been  reviewed).  Preliminary  ratings  are  developed  at  the  same  time 
as  the  preliminary  findings,  typically  during  the  afternoon  of  the  third 
on-site  appraisal  day.  The  advantage  of  developing  preliminary  ratings 
at  this  time  is  that  the  need  for  additional  information  may  be  brought  up 
while  developing  the  ratings.  There  is  a  further  opportunity  to  get 
additional  information  during  review  sessions  of  the  preliminary 
findings  with  the  systems  engineering  leads. 

Preliminary  ratings  are  a  synthesis  of  all  of  the  primary  data  collected. 
The  appraisal  team  must  determine  if  the  base  or  generic  practice  is 
performed  based  upon  the  data  shown  on  the  DTS.  Conflicting  data 
must  be  resolved  by  appraisal  team  consensus. 


The  final  ratings  are  developed  prior  to  developing  the  final  findings. 
They  serve  as  a  starting  point  for  developing  the  final  findings.  There  is 
generally  a  close  correlation  between  the  final  findings  and  the  process 
areas  with  a  low  rating  profile. 

Information  from  the  review  sessions  of  the  preliminary  and  final 
findings  is  used  to  adjust  the  ratings  as  needed. 


The  rating  algorithm  for  determining  a  capability  level  in  a  process  area 
is  the  appraised  entity  performing  100%  of  the  base  or  generic  practices 
in  the  respective  level.  In  addition,  a  higher  level  can  be  achieved  only 
if  a  lower  level  is  also  fulfilled. 

Performance  of  100%  of  the  base  or  generic  practices  is  estimated  for 
the  appraised  entity  as  a  whole  only  at  the  preliminary,  draft,  and  final 
ratings  steps.  As  explained  in  further  detail  in  the  DTS  Summary  in 
Appendix  C,  preliminary,  draft,  and  final  findings  and  the  rating  profile 
represent  a  summary  of  all  of  the  data  collected  over  the  appraisal  on-site 
period. 


continued  on  next  page 
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3.6  Developing  the  Rating  Profile,  Continued 


Possible  rating 

presentation 

formats 


Example  pie 
chart 


The  following  formats  for  presenting  the  rating  profile  have  all  been 

used  in  different  appraisal  contexts. 

•  Pie  chart:  number  of  PAs  at  specific  levels  (works  well  for 
organization  primarily  in  the  level  2-3  maturity  range). 

•Bar  chart:  score  for  each  PA  shown  on  a  bar  graph  that  allows  the 
option  to  use  quartiles  for  PAs  below  level  1  (good  option  for  the 
organization  in  the  0-2  maturity  range). 

•  Tabular  format. 


Profile-Pie  Chart 


PAs  at  level  2 


Figure  3-4.  Pie  Chart  Example. 


continued  on  next  page 
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3.6  Developing  the  Rating  Profile,  continued 


Example  bar 
chart 


&> 

« 

< 

O 

o 

o 


&yppifers 
Trainmg 
&ipport  £nwo?5?ncrit 
Product  Evo^utio^? 
Process  improvement 
De^ne  Process^ 
Planning^ 
Tradong 
Risk  !^<{anagement 
Con^gu-atiorj  iWgfnt, 
Quality 
vedfy&vaficsate^ 
Customer  Needs 
integrate 
OiSDpSftCS^ 
Ardijtecttjre 
Reguirernents 
Anatyze  Solutions 


E 


Capability  Levels 


Figure  3-5.  Bar  Chart  Example. 
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3.6  Developing  the  Rating  Profile,  continued 


Example  tabular 
format 


PA  Title 

Rating 

Analyze  Candidate  Solutions 

1 

Develop  Functional/Performance  Requirements 

2 

Develop  Physical  Architecture 

2 

Integrate  Disciplines 

0 

Integrate  System 

2 

Understand  Customer  Needs  &  Expectations 

1 

Verify  &  Validate  System 

2 

Ensure  Quality 

1 

Manage  Configurations 

3 

Manage  Risk 

1 

Monitor  &  Control  Technical  Effort 

1 

Plan  Technical  Effort 

2 

Define  Organization's  Systems  Engineering  Process 

0 

Improve  Organization's  Systems  Engineering  Processes 

1 

Manage  Product  Line  Evolution 

0 

Manage  Systems  Engineering  Support  Environment 

1 

Manage  Systems  Engineering  Training 

2 

Coordinate  with  Suppliers 

1 

Table  3-4.  Tabular  Format  Example. 


continued  on  next  page 
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3.7  Developing  Findings 


Introduction 


Exploratory 

questions 


The  appraisal  findings  are  a  key  product  of  the  appraisal.  They  are  the 
result  of  synthesizing  all  of  the  data  collected  throughout  the  on-site 
phase,  along  with  the  questionnaire  responses.  Findings  are  limited  to 
approximately  seven,  so  that  the  organization  is  left  with  a  number  of 
findings  that  is  manageable  in  forming  or  continuing  an  improvement 
effort. 


Exploratory  questions  are  developed  to  explore  the  reason  for  any 
inconsistencies  in  the  questionnaire  responses,  and  are  the  first  step  in 
eliciting  supporting  or  conflicting  information  on  the  performance  of 
base  or  generic  practices.  Exploratory  questions 

•  Should  be  limited  to  approximately  50  per  systems  engineering  lead, 
and  should  be  prioritized  due  to  interview  time  limitations. 

•  Should  be  linked  to  the  individual  process  areas  in  order  to  maintain 
traceability  to  the  model,  and  facilitate  the  data  management  that  needs 
to  occur  throughout  the  on-site  phase. 

•  Are  typically  a  mix  of  some  specific  questions  that  are  designed  to 
address  inconsistencies  on  a  specific  project,  or  general  questions 
aimed  at  possible  consistent  misinterpretations  of  the  questionnaire. 

Information  on  the  performance  of  base  or  generic  practices  is  recorded 
on  the  DTS  after  each  of  the  interviews  with  the  systems  engineering 
lead. 


continued  on  next  page 
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3.7  Developing  Findings,  Continued 


Preliminary 

findings 


Process  for 
developing  draft 
findings 


The  preliminary  findings  are  a  synthesis  of  the  primary  issues  that 
appraisal  team  members  have  collected  from  both  the  systems 
engineering  leads  and  the  practitioner  groups.  Preliminary  findings 
include  both  strengths  and  weaknesses. 

The  preliminary  findings  are  reviewed  with  the  systems  engineering 
leads  to  confirm  that  the  findings  are  true  for  their  project,  and  that  the 
findings  are  tme  for  the  organization  based  upon  their  knowledge.  The 
primary  objective  of  this  set  of  sessions  is  to  validate  the  synthesized 
comments. 

Feedback  from  the  systems  engineering  leads  also  helps  in  the 
prioritization  of  the  data.  The  number  of  times  that  a  preliminary  finding 
is  supported  or  contradicted  helps  the  appraisal  team  determine  high- 
priority  items  in  the  next  synthesis  step  to  develop  the  draft  findings. 

An  additional  objective  of  the  feedback  sessions  is  to  obtain  more 
supporting  or  conflicting  information  that  is  used  to  establish  the  rating. 
Additional  questions  may  be  inserted  to  address  uncertainties  that  the 
appraisal  team  may  have  with  respect  to  the  rating  profile. 


The  draft  findings  are  the  unreviewed  version  of  the  final  findings. 
Confirmed  preliminary  findings  are  clustered  using  a  technique  such  as 
affinity  diagram,  and  a  set  of  5  to  10  themes  or  underlying  factors  are 
derived  to  form  the  draft  findings.  Draft  findings  are  presented  via 
briefing  charts  in  the  format  of  finding,  cause,  and  consequence. 

The  draft  and  final  findings  typically  fall  into  one  of  three  categories: 

•  General  barriers  to  the  next  level. 

•  Weaknesses  in  the  base  practices. 

•  Weaknesses  in  the  generic  practices. 

The  draft  findings  are  presented  to  the  systems  engineering  leads  and 
practitioners  to  get  validation  of  the  draft  findings,  to  give  the  appraisal 
team  leader  a  chance  to  diy-run  the  presentations,  and  to  allow  a  fomm 
for  the  practitioners  to  refine  the  findings. 


continued  on  next  page 
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3.7  Developing  Findings,  continued 


Finding,  cause, 
consequence 


Final  findings 


Once  a  set  of  findings  areas  is  identified,  the  team  should  brainstorm  the 
findings,  causes,  and  consequences.  The  main  findings  should  be 
carefully  worded  to  reflect  what  the  team  has  actually  heard  from  the 
participants.  The  finding  is  usually  a  single  observation;  it  may  be 
thought  of  as  a  characterization  of  a  symptom.  An  example  of  a  finding 
is 

System  engineering  plans  do  not  realistically  reflect  the  needs  of  the 
projects. 

Causes  are  observations  that  support  the  central  finding,  and  many 
indicate  potential  causes  of  the  finding.  An  example  of  a  cause  is 

Estimates  are  not  based  on  available  historical  data. 

Consequences  list  the  probable  results  of  the  finding.  An  example  of  a 
consequence  is 

Cost  &  schedule  overruns 

The  causes  and  consequences  are  often  taken  directly  from  the 
preliminary  findings.  The  causes  are  an  opportunity  to  employ  the 
phrasing  heard  during  the  practitioner's  sessions.  On  the  other  hand, 
consequences  primarily  target  the  sponsor  and  should  reflect  his/her 
perspective.  Use  of  organizational  objectives  or  goals  and  the  risks  to 
meeting  those  goals  is  recommended  as  part  of  the  consequences. 

During  this  initial  step  do  not  get  bogged  down.  If  the  team  cannot 
agree  on  wording,  leave  it  for  the  next  step.  At  the  end  of  this  step,  the 
findings  have  been  identified  along  with  a  set  of  causes  and 
consequences. 


Final  findings  are  an  edited  version  of  the  draft  findings  based  upon 
comments  from  the  practitioners  and  systems  engineering  leads.  These 
are  used  to  develop  the  appraisal  report  and  recommendations. 

A  sample  final  findings  briefing  is  included  in  Appendix  B  for 
reference. 


SECMM-94-06ICMU/SEI-96-HB-004  v1 .1 


3-27 


Appendices 


Introduction 


In  these 
appendices 


These  appendices  provide  support  materials  that  organizations  using 
SAM  may  find  helpful  in  constructing  appraisal  artifacts  and  training 
team  members.  The  appendices  can  be  acquired  from  the  SEI  and  SPC 
FTP  sites  as  separate  electronic  documents  for  the  appraisers. 

To  access  electronic  documents  from  the  SPC  FTP  site 
Type  ftp  ftp.  software,  o  rg 
You  will  be  prompted  to  log  in 
Type  anonymous 

You  will  be  prompted  for  a  password 

Type  in  your  email  address  in  the  form  name© site 

[for  statistical  purposes  only] 

Type  cd  secmm 
Type  cd  public 
Type  dir 
Type  binary 

Type  get(  space  )filename 
Type  quit  to  finish 


Topic 

See  Page 

Appendix  A:  Template  for  SAM  Opening  Briefing 

A-3 

Appendix  B:  Template  for  SAM  Final  Findings 

Briefing 

A-15 

Appendix  C:  Data  Tracking  Sheet  and  Instructions 

A-27 

Appendix  D:  Sample  Schedules  for  the  On-Site  Week 

A-30 

Appendix  E:  SAM  Training  Support 

A-41 

Appendix  F:  Site  Coordination  Checklist 

A-64 

Appendix  G:  Approved  SAM  Requirements 

A-71 

Appendix  H:  Traceability  Matrix  to  SEI  CMM 

Appraisal  Framework 

A-75 

Appendix  I:  References 

A-88 

Appendix  J:  SAM  Questionnaire 
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Appendix  A:  Template  for  SAM  Opening  Briefing 


Introduction 


The  opening  briefing  template  provides  ideas  on  how  to  present  the 
basic  concepts  of  and  the  SE-CMM  to  the  appraisal  participants. 
However,  appraisers  are  welcome  to  design  whatever  materials  suit  their 
style/needs.  The  basic  purpose  of  the  opening  briefing  and  suggested 
contents  are  found  in  Section  2.2.1. 

Also  included  in  this  appendix  are  some  of  the  basic  information  charts 
included  in  the  SE-CMM  Project  Overview  briefing.  For  audiences 
who  are  somewhat  unfamiliar  with  the  model,  these  charts  or  similar 
ones  may  be  of  use  as  backup  charts  for  the  opening  briefing  to  help 
answer  questions  about  the  model  itself. 


continued  on  next  page 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  continued 


SE-CMM 


Systems  Engineering  Improvement 


<company> 


SE-CMM  Appraisal 


<date> 

@CMM  1996.  This  work  Is  a  collaborative  effort  of  EPIC:  FAA,  GTE  Gov't.  Systems,  Electronic 
Boat  Division  of  General  Dynamics,  Hughes  Corporation,  Lockheed-Martin  Corp.,  Loral  Corp., 
NIST,  Office  of  Undersecreatary  of  Defense  for  Acquisition  and  Technology,  SEI,  SPC,  and 
Texas  Instruments,  Inc. 


Process  Maturit 


Benefits  of  process  maturity 

—  increased  accuracy  in  predicting  results 
—Reduced  variability  in.  expected  outcome 

—  Improved  productivity 
Organizational  Agility 

— New  T  echnology 

—  New  Markets 


People  and  Technology 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  continued 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  Continued 


Extra  brieflng 
charts 


The  following  set  of  charts  may  be  useful  as  backup  charts  if  the 
audience  is  not  sufficiently  familiar  with  the  SE-CMM  model  itself. 


SE  -  CMM  - 

industiy  —  Academia  —  Govemment - ►  Systems  Engineering  Improvement 

What  is  Systems  Engineering? 

(Many  definitions  of  systems  engineering  have  been  published.  The  authors  chose  to 
use  the  definition  of  systems  engineering  from  AFM  770-78:) 

Systems  Engineering  is  defined  as  the  selective 
application  of  scientific  and  engineering  efforts 
to: 

1.  Transform  an  operational  need  into  a  description  of  a  system  configuration  which 
best  satisfies  the  operational  need  according  to  the  measures  of  effectiveness; 

2.  Integrate  related  technical  parameters  and  ensure  compatibility  of  ail  physical, 
functional,  and  technical  program  interfaces  in  a  manner  which  optimizes  the  total 
system  definition  and  design; 

3.  Integrate  the  efforts  of  all  engineering  disciplines  and  specialities  into  the  total 
engineering  effort. 


SE-CMM 


Indussy  —  Academia  —  Govemme.m 


►  Systems  Engineering  impro  vement 


►  Strategic  Focus 
‘  Sterket  Pull  vs. 
Technology  Push 
‘  Contract  vs. 

Market  Driven 

Technoiogy/Method 

Support 


SE-CMM 


Guidance 

•  Focus  Area 
(Domain) 

•  Capability 

•  Support 


Organization’s 
SE  Process 
Development 


Design 

Development 

>  Validation 
and 

Verificadon 


continued  on  next  page 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  continued 


SE-CMM 

Industry  —  Ac 


Academia  —  Government 


Systems  Engineering  Improvemem 


Orwmfaatiwi’ 

^Proc#s$. 


SEProc^scMi; 


Process  Area  List 


,  Systems  Engineering  Improvement 


Understand  customer 
needs  and  expectations 

Derive  and  allocate 
requirements 

Analyze  candidate 
solutions 

Evolve  system  architecture 
Integrate  system 
Verify  &  validate  system 
Integrate  disciplines 


Project 


Manage 

configurations 

Plan  technical  effort 

Monitor  and  control 
technical  effort 

Manage  risk 
Ensure  quality 


Organization 


uenne  org 
systems  engineering 
process 

Manage  systems 
engineering  support 
environment 

Provide  ongoing  skills  and 
knowledge 
Manage  product  line 
evolution 

Improve  organization’s 
systems  engineering 
processes 

Coordinate  with  suppliers 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  Continued 


Industry  —  Academia  —  Government 


Systems  Engineering  Improvement 


“Typical”  Process  Area 

PA  05:  Integrate  System 

•  The  purpose  of  Integrate  System  is  to  ensure 
that  system  elements  will  function  as  a 
whole.  This  primarily  involves  identifying, 
defining,  and  controlling  interfaces,  as  well 
as  verifying  system  functions  that  require 
multiple  system  elements. 


Systems  Engineering  Improvement 


‘‘Typical  Generic  Practice” 

I  Generic  Practices 

\l.O  Performed  Level 

r  Common  Feature  1:  Base  Practices  are 
performed 

d  1.1.1  Perform  the  process.  Perform  a  process 
\  that  implements  the  base  practices  of  the 
/  process  area  to  provide  work  products  and/or 
f  sen/ices  to  a  customer. 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  continued 


I 


I 


Industry  —  Academia  —  Government 

Relationships 


Systems  Engineering  Improvement 


Capability  Level  2 


SE-CMM _ 

Industry  —  Academia  —  Government 


Systems  Engineering  Improvement 


Roadmap  for  Improvement 


NOT 

PERFORMED 


PERFORMED 

NFORMAaV 


Base  practices 
performed 


PLANNED  & 

TRACKED 

•  Cofr> milting  to 
perform 

•  Planning 
performance 

•  Disciplined 
performance 

•Tracking 

performance 

•Verffytng 

performance 


WEa-DEFINED 

Defining  a 
standard  process 
‘Tailoring  the 
standard  process 
•  Using  data 
>  Perform  the 
defined  process 


5 


iQUANTiTATIVELY 

CONTROLLED 


’  Establishing 
measurable  quality 


'  Determining 
process  capability  to 
achieve  goals 
■  Objectively 
managing 
performance 


CONTINUOUSLY 

MPROVING 

>  Establishing 

quantitative 

process 

effectiveness 

goals 

■  Improving  process 
effectiveness 


Common 

Features 


continued  on  next  page 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  continued 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing 


Introduction  The  template  for  the  final  findings  briefing  provides  ideas  on  how  to 

present  the  findings  resulting  from  the  on-site  period  of  the  appraisal  to 
the  appraisal  participants.  Appraisers  are  welcome  to  design  whatever 
materials  suit  their  style/needs.  The  basic  purpose  of  the  final  findings 
briefing  and  suggested  contents  are  found  in  Section  2.2.14. 

continued  on  next  page 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 


Findings  Development  Process, 
continued 

•  Findings  Criteria 

-  We  heard  it 

-  No  sweeping  statements 

-  Only  issues  with  potential  recommendations 

-  Appraisal  team  consensus 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 


Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 


General  ‘<capability  level>’ 
Barriers 

Findings: 

-  <findings  which  generally  apply  across  the  process  areas 
which  create  a  barrier  to  the  next  capability  level> 

Consequences: 

-  <process  capability-related  consequences  of  find!ngs> 


Next  Steps 


Findings  and  Recommendations  Report 

Action  Plan 

improvements! 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 


If  you  always  do  what  you've 
always  done 

you'll  always  get  what 
you've  always  gotten! 


continued  on  next  page 
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Appendix  C:  Data  Tracking  Sheet  and  Instructions 


Introduction 


Symbols  used  in 
the  DTS 


Structure  of  the 
DTS 


The  data  tracking  sheet  (DTS)  is  the  prima^  tool  used  throughout  the 
on-site  period  to  consolidate  and  track  the  information  being  gathered 
from  the  different  data  sources  used  in  the  appraisal.  It  is  a  key  input 
into  the  ratings  and  findings  development  process,  and  when  used 
effectively,  adds  greatly  to  the  team's  ability  to  understand  where 
sufficient  data  have  been  obtained,  and  where  additional  information  is 
still  needed  about  the  practice  of  the  base  and  generic  practices  within 
the  appraised  entity.  The  DTS  is  referred  to  throughout  the  process 
elements  of  the  on-site  period  described  in  Chapter  2. 


Data  in  the  DTS  are  recorded  primarily  via  the  use  of  four  symbols: 

1  or  +  Indicates  that  information  was  heard  or  otherwise  obtained  that 
strengthens  or  corroborates  the  view  that  the  practice  under 
consideration  is  being  performed. 

0  or  -  Indicates  that  information  was  heard  or  otherwise  obtained  that 
weakens  or  opposes  the  view  that  the  practice  under 
consideration  is  being  performed. 

Blank  or  ?  Indicates  that  information  was  heard  or  otherwise  obtained  in 
relation  to  the  practice,  but  the  nature  of  the  information  was 
such  that  questions  related  to  the  performance  of  the  practice 
have  not  been  answered. 

NA  Indicates  that  the  practice  is  considered  not  to  be  applicable  to  the 
appraised  entity. 

The  use  of  these  symbols  is  explained  in  the  process  element 

descriptions  in  the  main  body  of  the  document,  and  illustrated  in  the 

blocks  below. 


The  DTS  is  composed  of  three  pages  per  process  area.  Each  page  lists 
the  short  titles  of  the  base  practices  for  that  PA  and  the  generic  practices 
for  all  five  capability  levels  as  the  rows  of  a  matrix.  The  colunms  of  the 
matrix  provide  space  to  record  the  gathering  of  corroborating  or 
opposing  information  in  relation  to  the  base  and  generic  practices  of  the 
SE-CMM.  Note  that  information  other  than  that  specifically  related  to 
the  model  is  likely  to  be  gathered  in  the  team  notes,  and  may  be  used  to 
create  findings  where  appropriate.  However,  these  data  are  not 
specifically  used  in  the  creation  of  the  rating  and  so  are  not  recorded  in 
the  DTS. 


continued  on  next  page 
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Appendix  C:  Data  Tracking  Sheet  and  Instructions,  continued 


DTS  Contents  Table  C-1  explains  each  of  the  columns  of  the  DTS: 


Column  Title 

Explanation 

Base/Generic 

Practices 

This  column  contains  a  listing  of  the  short  titles 
of  the  base  practices  for  the  listed  PA,  as  well 
as  all  the  generic  practices. 

Questionnaires  A-G 

These  columns  record  the  translation  of  the 
yes/no  responses  from  the  questionnaires  of  the 
respondents  into  -I-/-/7/NA  symbols. 

Interviews  A-D 

It  is  assumed  that  a  maximum  of  four  projects 
will  be  appraised  in  a  single  appraisal.  These 
columns  provide  space  for  recording 
impressions  from  the  initial  interviews  with 
the  SE  leads. 

Practitioner 

Interviews 

These  columns  provide  space  for  recording 
impressions  from  the  practitioner  interviews, 
of  which  there  are  usually  three  focus  groups. 

Preliminary  Rating 

This  the  first  estimate  of  the  rating  of  each 
base  or  generic  practice.  Each  team  member 
uses  the  pattern  of  corroborating  and 
opposing  information  to  formulate  an  initial 
opinion  of  the  performance  of  that  practice  in 
the  appraised  entity  as  a  whole. 

Preliminary  Findings 
Review  A-D 

These  columns  provide  space  for  recording 
impressions  from  the  preliminary  findings 
review  with  the  SE  leads. 

Draft  Rating 

This  column  is  where  the  preliminary  rating  is 
refined  with  data  gathered  via  the  preliminary 
findings  review,  and  is  the  rating  that  will  be 
correlated  with  the  findings  development. 

Findings  Reviews 

These  columns  provide  space  for  recording 
information  gathered  during  the  draft  findings 
reviews  by  the  SE  leads  and  practitioners. 

Final  Rating 

This  is  the  final  rating  for  each  base/generic 
practice  upon  which  team  has  reached 
consensus,  and  is  the  basis  for  the  profile  that 
is  presented  in  the  final  findings  briefing. 

Table  C-1.  DTS  Contents. 


continued  on  next  page 
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Appendix  C:  Data  Tracking  Sheet  and  Instructions,  Continued 


Diagram  of  an 
example  DTS 


A  DTS  can  be  created  in  a  spreadsheet,  database,  or  other  tool 
compatible  with  the  appraiser’s  environment.  The  following  diagram 
provides  an  example  of  a  DTS  created  in  a  spreadsheet  environment. 


Ratinasl 


Questionnaire 


Prelimina 


PA  01:  Analyze 
Candidate  Solutions 


Draft  Final 


SE  Leads  Feedbk 


Establish  Evaluation 
Criteria  _ 


Define  Analysis  Approach  | 
Identify  Additional  Alternatives 


1.6  Capture  Results 


Generic  2.1.1  Allocate  resources 

2.1.2  Assign  responsibilities 


Ensure  trainin 


Plan  the  process _ 

Use  plans,  standards,  and 

procedures _ 

Do  configuration 
manaaement 


Veri 


Audit  work  products 


Track  with  measurement 
Take  Corrective  Action 


Standardize  the  process 
Tailor  the  standrad  process 


2.3.1 


2.3,2 


2.4.1 

2.4.2 


3.1.1 

3.1.2 


“Si 


3.2.1 

3.2.2 

Use  a  well-defined  process 
Perform  defect  reviews 

3.2.3 

Use  well-defined  data 

4.1.1 

Establish  quality  goals 

4.2.1 

Determine  process 
capability 

4.2.2 

Use  process  capability 

5.1.1 

Establish  process 
effectiveness  goals 

5.1.2 

Continuously  improve  the 
standard  process 

5.2.1 

Perform  casual  analysis 

Figure  C-1.  DTS  Example  Diagram. 


Appendix  D:  Sample  Schedules  for  the  On-Site  Week 


Introduction 


Five  Day 
SAM 


Sample  on-site  schedules  and  labor  temples  are  provided  in  this  section. 

The  bases  of  the  samples  are  an  appraisd  of  three  projects  with  three  sets  of 
practitioner  groups. 

It  should  be  noted  that  a  series  of  30  minutes  breaks  are  embedded  into  the 
schedules.  These  breaks  can  be  used  by  the  team  as  situations  and 
requirements  dictate. 


The  following  figures  and  table  provide  an  overview  of  a  typical  five  day 
on-site  appraisal  period.  The  process  flow  is  demonstrated  in  Figure  D-1. 


Monday  Tuesday  Wednesday  Thursday  Friday 


Figure  D-1. 

Typical  Five  Day  Appraisal  Process  Flow. 
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Appendix  D:  Sample  Schedules  for  the  On-Site  Week,  Continued 


Typical  Five  Day 
SAM  Schedule  & 
Labor 

Requirements 


Start 

Finish 

Dur. 

Description 

8:00  AM 

9:30  AM 

1:30 

Opening  Meeting 

10:00  AM 

12:00  PM 

2 

Brief  Team 

12:00  PM 

1:00  PM 

2 

Lunch 

1:00  PM 

5:00  PM 

2 

Analyze  Questionnaire  and  Develop  Response 

8:00  AM 

9:00  AM 

1 

Interview  SE  Lead  #1 

9:30  AM 

10:30  AM 

1 

Interview  SE  Lead  #2 

11:00  AM 

12:00  PM 

1 

Interview  SE  Lead  #3 

12:00  PM 

1:00  PM 

1 

Lunch 

1:00  PM 

3:00  PM 

2 

Interview  Practitioner  Group  #1 

3:30  PM 

4:30  PM 

1 

Data  Consolidation 

Wednesday 

8:00  AM 

10:00  AM 

2 

Interview  Practitioner  Group  #2 

10:30  AM 

12:30  PM 

2 

Interview  Practitioner  Group  #3 

12:30  PM 

1:30  PM 

1 

Lunch 

1:30  PM 

2:30  PM 

1 

Data  Consolidation 

3:00  PM 

6:00  -  9:00  PM 

3-6 

Develop  Preliminary  Findings 

8:00  AM 

9:30  AM 

1 

Review  Preliminary  Findings  with  SE  Lead  #1 

9:30  AM 

10:30  AM 

1 

Review  Preliminary  Findings  with  SE  Lead  #2 

11:00  AM 

12:00  PM 

1 

Review  Preliminary  Findings  with  SE  Lead  #3 

12:00  PM 

1:00  PM 

1 

Lunch 

1:00  PM 

3:00  PM 

1 

Develop  Draft  Rating 

3:00  PM 

7:00-  11:00  PM 

4-8 

Draft  Finding  Ratings 

Friday 

8:00  AM 

9:00  AM 

1 

Present  Draft  Findings  to  Practitioners 

9:30  AM 

10:30  AM 

1 

Present  Draft  Findings  to  SE  Leads 

10:30  AM 

12:30  PM 

2 

Adjust  Draft  Findings 

12:30  PM 

1:30  PM 

1 

Lunch 

2:00  PM 

3:30  PM 

1:30 

Present  Final  Briefing 

3:30  PM 

4:30  PM 

1 

Conduct  Executive  Session 

4:30  PM 

5:30  PM 

1 

Wrap-up 

Figure  D-2. 

Typical  Five  Day  Appraisal  Schedule. 
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Appondix  Dl  Sdmpi@  Schodulos  for  th©  On~SitG  WGOk,  Continued 


Role 

Recommended 
number  of 
people 

Hours 

per 

person 

Total  Hours  for 
this  role 

Facilitator 

2 

60 

120 

Appraisal  team 
member  (in 
addition  to 
facilitators 

4-6 

50 

200-300 

Systems 

engineering  leads 

3 

(1  per  project  @  3 
projects) 

7 

21 

Practitioners  from 

across 

organization 

30 

(3  groups  of  10) 

6 

180 

TOTAL 

521  -621 

Table  D-1.  Sample  Labor  Requirements  for  a  Five  Day  Appraisal 

Two+Five 

Day  SAM  The  following  fibres  and  table  provide  an  overview  of  a  typical  Two+Five 
day  on-site  appraisal  period.  The  process  flow  is  demonstrated  in 
Figure  D-3 


Visit  #1 


Day#1 


Day  *2 


Visit  #2 


Monday 


Tuesday 


Wednesday  Thursday 


Friday 


Figure  D-3. 

Typical  Two+Five  Day  Appraisal  Process  Flow. 
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Appendix  D:  Sample  Schedules  for  the  On-Site  Week,  continued 


Typical 

Two+Five  Day 
SAM  Schedule 
&  Labor 
Requirements 


Start 

Finish 

Dur. 

Description 

Day  #1 

8:00  AM 

9:00  AM 

1 

Appraisal  Team  Introductions 

9:00  AM 

12:00  PM 

3 

Present  SE-CMM  Model 

12:00  PM 

1:00  PM 

1 

Lunch 

1:00  PM 

5:00  PM 

4 

Present  SAM 

Day  #2 

8:00  AM 

10:00  AM 

2 

Administer  Questions 

10:00  AM 

11:00  AM 

1 

Enter  Data  in  Data  Tracking  Sheet 

11:00  AM 

12:00  PM 

1 

Reveiew  Questionnaire  Responses 

12:00  PM 

1:00  PM 

1 

Lunch 

1:00  PM 

5:00  PM 

4 

Analyze  Questionnaire  and  Develop  Response 

Figure  D*4.  Typical  Two+Five  Day  Appraisal  Schedule 


Role 

Recommended 
number  of 
people 

Hours 

per 

person 

Total  Hours  for 
this  role 

Facilitator 

2 

76 

152 

Appraisal  team 
member  (in 
addition  to 
facilitators 

4-6 

66 

264-396 

Systems 

engineering  leads 

3 

(1  per  project  @  3 
projects) 

9 

27 

Practitioners  from 

across 

organization 

30 

(3  groups  of  10) 

8 

240 

TOTAL 

683-  815 

Table  D-2.  Sample  Labor  Requirements  for  a  Two+Five  Day 

Appraisal. 
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Appendix  D:  Sample  Schedules  for  the  On-Site  Week,  continued 


SE  or  Project  Lead’s  Appraisal  Schedule 


Name: 


Individual:  Lead  Systems  Engineer 

Appraisal  participant  identified  as  having  responsibility  for  the  systems  engineering 
aspects  of  a  project.  The  systems  engineering  lead  should  have  a  broad  knowledge  of 
the  full  life  cycle  of  the  product  development  cycle. 

Labor  7  Hours  per  Person  total 

Requirements:  1  Person  Per  Project 

Overview:  During  the  SE-CMM,  the  appraisal  team  will  be  using  the  responses  to  the  SAM 

(Systems  Engineering  Capability  Maturity  Model  Appraisal  Method)  Questionnaire  to 
develop  a  series  of  exploratory  questions.  The  intended  use  of  these  questions  is  to 
develop  a  better  understanding  of  the  organization.  Lead  SEs  are  interviewed  to  clarify 
questionnaire  responses.  The  interview  responses  of  the  lead  systems  engineers  and  the 
practitioner  focus  groups  are  combined  to  develop  preliminary  findings.  Preliminary 
findings  are  reviewed  and  prioritized  by  lead  SEs.  The  findings  are  then  refined  into 
draft  findings.  The  draft  is  then  reviewed  by  both  the  lead  systems  engineers  and  the 
practitioners  to  develop  the  final  findings.  These  are  presented  with  ?  to  the  sponsor  in 
?  session  with  lead  SEs  and  practitioners. 

SE-CMM 

Events: 

•  Complete  the  SAM  questionnaire 

Complete  the  SAM  Questionnaire  prior  to  the  SE-CMM  Appraisal  on-site  week 


Room: 


Date: 


Time: 


•  Participate  in  the  Opening  Briefing 
Attend  Opening  Briefing 


Room: 


Date: 


Time: 


Participate  in  a  series  of  question  and  answer  sessions 

Based  upon  the  questionnaire,  the  appraisal  team  will  develop  a  set  of 
exploratory  questions  for  each  of  the  systems  engineering  lead,  and  ’’listen  fors”. 
The  questions  will  be  proposed  to  the  lead  systems  engineers  and  the  responses 
recoit^ 

The  appraisal  team  will  consolidate  the  answers,  along  with  the  practitioners 
responses,  into  preliminary  findings 


Room: 


Date: 


Time: 
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Appendix  D:  Sample  Schedules  for  the  On-Site  Week,  continued 


•  Provide  feedback  on  the  validity  of  the  findings 

Each  of  the  preliminary  findings  are  presented  to  each  lead  systems  engineer  to 
assess: 

>  Is  it  true  for  his/her  project 

>  Is  it  true  for  the  organization 

Additional  questions  may  be  asked  for  clarification  or  for  follow-on 


Room: 


Date: 


Time: 


•  Provide  feedback  on  the  draft  findings 

Provide  feedback  on  whether  the  appraisal  team  captured  what  is  happening  in 
the  organization 

Assist  crafting  of  draft  into  final  findings  for  maximum  impact 


Room: 


Date: 


Time: 


•  Participate  in  the  final  briefing 

Be  prepared  to  support  findings  to  sponsor 

Consider  activities  that  could  mitigate  findings  or  improve  practice  in  areas 
identified 


Room: 


Date: 


Time: 
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SE  or  Related  Practitioner’s  Appraisal  Schedule 

Name: 


Individual :  Practitioners 

Individuals  who  perform  or  support  the  systems  engineering  process  (directly  or 
indirectly,  e.g.,  training,  QA,  CM,  marketing,  logistics,  field  support). 

Labor  5  Hours  per  Person 

Requirements:  4-10  People  Per  Group 

Overview:  Through  open-ended  discussion  with  different  types  of  practitioners,  corroborating  or 

contradictory  data  is  gathered  on  SE,  program  and  organizational  practices.  The 
responses  of  the  individual  practitioners  and  those  of  the  lead  systems  engineers  are 
combined  to  develop  preliminary  findings.  The  preliminary  findings  are  then  refined 
into  draft  form  with  feedback  from  the  lead  systems  engineers.  The  draft  is  reviewed  by 
both  the  lead  systems  engineers  and  the  practitioners  to  develop  the  final  findings. 

SE-CMM 

Events: 

•  Participate  in  Opening  Briefing 

Listen,  and  ask  clarifying  questions  about  method  or  goals. 


Room: 


Date: 


Time: 


Participate  in  focus  groups. 

A  free- form  discussion  centered  around  the  question  "what  works  or  doesn't  work 
well  in  the  systems  engineering  process?" 

The  facilitator  may  provide  minimal  direction  in  appropriate  process  areas 
The  responses  are  recorded 

The  appraisal  team  will  consolidate  the  information  gathered,  along  with  the  lead 
systems  engineers’  data,  into  preliminary  findings 


Room: 


Date: 


Time: 


•  Provide  feedback  on  the  draft  findings 

Provide  feedback  on  whether  the  appraisal  team  captured  what  is  happening  in 
the  organization 


Room: 


Date: 


Time: 


•  Participate  in  the  final  briefing 

Be  prepared  to  support  findings  to  the  sponsor 


Room: 


Date: 


Time: 
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Appendix  D:  Sample  Schedules  for  the  On-SIte  Week,  continued 
Sponsor’s  Appraisal  Schedule 


Name: 

Individual: 


Labor 

Requirements: 

Overview: 


SE-CMM 

Events: 


Appraisal  Sponsor 

The  sponsor  is  the  person(s)  with  authority  to  direct  and  pay  for  an  appraisal.  The 
sponsor  has  a  leadership  role  in  changing  the  practice  of  the  organization.  Visible 
endorsement  of  the  appraisal's  goals  and  its  results  during  opening  session,  and 
acceptance  of  results  and  commitment  to  improvement  at  final  briefing  are  critical  to 
the  appraisal  success. 

4  Hours  per  Person 
1  or  2  People 

The  sponsor  relates  the  appraisal  to  business  objectives  and  the  improvement  efforts  of 
the  organization.  An  initial  pre- appraisal  session  sets  goals  and  context  or  motivation 
for  performing  the  appraisal.  At  the  opening  briefing,  the  sponsor  visibly  encourages 
open  and  candid  participation  and  provides  his/her  motivation  for  the  appraisal.  At  the 
final  presentation,  the  sponsor  accepts  the  results,  notes  that  there  is  plenty  to  do, 
thanks  the  participants  for  their  contributions,  and  may  indicate  how  the  appraisal 
results  will  be  used  in  the  organization's  improvement  plan. 


•  Establish  appraisal  goals,  scope,  resources,  and  approximate  schedule 

Meet  with  team  leader  or  facilitator  and  perform  pre-on-site  planning  activities 


Room:  sponsors'  office  Date: 


Time: 


•  Participate  in  Opening  Briefing. 

Address  appraisal  participants  as  to  the  purpose  and  scope  of  appraisal,  outline 
business  objective  and  organizational  improvement  efforts. 

Encourage  open,  candid  dialogue  by  all  participants 

Indicate  interest  in  outcomes 

Assure  participants  of  safety  regardless  of  results. 


Room: 


Date: 


Time: 


•  Participant  in  Final  Findings  briefing 

Accept  results  after  briefing  (say,  "I  hear  you") 

Thank  participants  for  effort  and  outcome 

Indicate  how  results  will  be  used  in  improvement  efforts  (e.g.,  action  plan  part 
in  improvement  efforts). 


Room: 


Date: 


Time: 
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•  Optional;  Executive  Session  (may  precede  final  session) 

Provide  an  opportunity  for  classifications  closer  to  detail  level  but  still  non- 
attributable  to  project(s)  or  individuals 

Provides  an  opportunity  to  ask  ancillary  questions  such  as  "what  was  your 
biggest  insight  not  in  the  briefing?"  to  each  team  member. 

•  Support  Report  Development,  Action  Planning,  and  Improvement 

Provide  resources  and  visible  support  for  recommendations,  and  action  planning 
and 

Provide  resources,  implementation  visible  interest  and  support  for  plan 
performance. 
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Appendix  D:  Sample  Schedules  for  the  On-Site  Week,  continued 


Appraisal  Team  Member’s  Schedule 

Appraisal  Team  Appraisal  team  members  are  expected  to  take  training  and  assist  with 
Member  appraisal  questionnaire  interviews  distribution,  collection  and  analysis 

as  requested  pre-on-site,  and  to  special  Sam  to  "whenever  done" 
Monday  through  Friday  of  on-site  week. 
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Appendix  D:  Sample  Schedules  for  the  On-Site  Week,  continued 
Appraisal  Team  Leader  Schedule 


Appraisal  Team  The  difference  between  ATL  and  other  team  members  is  that  the  ATL 
Leader  assumes  authority  by  welcoming  participants  and  acting  a  master  of 

ceremonies  for  the  opening  meeting,  she  or  he  also  delivers  the  draft 
findings  and  final  findings  presentations. 
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Appendix  E:  SAM  Training  Support 


Introduction 


The  SE-CMM  Project  has  not  committed  to  producing  training  materials 
for  SAM.  However,  the  project  recognizes  that  some  training  beyond 
similar  organizational  appraisal  methods  is  necessary  to  prepare 
appraisal  team  members.  Many  of  the  materials  provided  in  the  SAM 
appendices  can  be  viewed  as  training  support  materials.  The 
information  sheets  provided  in  this  appendix  summarize  the  steps  of  the 
on-site  period  for  SAM  and  provide  an  easy  reference  for  appraisers  to 
determine  the  sequence  of  events. 
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Appendix  E:  SAM  Training  Support,  continued 

Conduct  Opening  Meeting 


Goals 


-  Visibly  demonstrate  senior  management  support  for  the  assessment  process. 

-  Introduce  principles  of  process  management. 

—  Review  schedule  for  the  assessment  period. 

'FScpqpiiSsr 

Diiratidn;  r  ; 


Wi 


'■••‘4 -7- 


Approach 

The  senior  site  manager  begins  this  meeting  by  welcoming  the  assessment  team  and  indicating 
management  support  for  the  assessment  process.  The  assessment  team  leader  delivers  a  brief 
presentation  on  process  management  and  the  assessment  process  flow.  Finally,  the  site 
coordinator  reviews  the  assessment  schedule,  reminding  everyone  of  when  and  where  they  are 
expected  to  participate. 

Senior  management  opening. 

-  Introduce  and  welcome  assessment  team. 

—  Indicate  support  for  assessment  and  process  improvement. 

-  Solicit  full  support  and  participation. 

G  Briefing  on  process  management  principles. 

Q  Briefing  on  assessment  process  flow. 

-  Stress  openness. 

-  Emphasize  confidentiality. 

G  Repeat  confidentiality  rules 

—  No  individuals  or  projects  named  in  results. 

-  They  may  not  disclose  comments  of  others  made  at  this  meeting. 

-  Team  will  take  notes.  All  notes  will  be  treated  as  confidential. 

G  Review  of  assessment  schedule. 

G  Question  and  answer  period. 
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Appendix  E:  SAM  Training  Support,  Continued 
Familiarize  Team  with  SAM 


Goals 


-  Introduce  the  team  to  the  assessment  process. 

-  Prepare  for  the  discussions  with  the  SE  leaders  and  practitioners. 


-  Review  the  answers  to  the  questionnaire. 


This  is  an  opportunity  for  the  assessment  team  to  begin  to  work  together.  The  assessment 
steps  are  presented  in  greater  detail  than  at  the  opening  meeting,  and  the  team’s  role  in  each  step 
is  clarified.  The  assessment  work  begins  with  a  review  of  the  answers  to  the  questionnaire 
along  with  the  exploratory  questions  for  the  project  leaders.  The  exploratory  questions  may  be 
tailored  where  special  conditions  are  identified. 

Approach 

□ 

□ 


□ 

□ 

□ 


Team  building  exercise  (optional). 

Review  of  assessment  steps. 

-  Explain  conduct  of  SE  leader  discussions. 

-  Explain  conduct  of  practitioner  discussions. 

Examine  exploratory  questions  and  answers  to  the  questionnaire. 
Review  assessment  schedule. 

Question  and  answer  period. 
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Appendix  E:  SAM  Training  Support,  continued 

Analyze  Questionnaire 


Goals 


-  Review  the  answers  to  the  questionnaire. 

-  Develop  exploratory  questions. 


-  Prepare  for  discussions  with  the  SE  leader. 


The  assessment  begins  with  a  review  of  the  answers  to  the  questionnaire.  Based  on  the 
answers  to  the  questionnaire,  the  team  develops  exploratory  questions.  Approximately  40-50 
questions  should  be  generated  for  each  project  lead.  The  questions  should  be  designed  to  elicit 
more  then  just  a  "yes/no"  response.  For  example,  questions  often  begin  with,  “Would  you 
please  describe  . . .”  When  developing  a  question  we  are  often  looking  for  certain  responses, 
e.g.,  “SE  management  plan.”  Note  these  words  as  “listen  fors,"  which  the  facilitator  can  use 
as  a  cue  to  ask  additional  questions  if  they  are  not  mentioned.  Some  questions  may  be 
accompanied  by  a  request  for  relevant  or  supporting  documents. 


Questions  should  be  used  to  refine  answers  or  explore  inconsistencies.  There  is  a  separate  set 
of  questions  for  each  SE  lead.  However,  there  is  usually  some  overlap.  Once  the  questions 
are  developed,  they  should  be  transcribed  onto  the  appropriate  form,  and  copies  should  be 
made  for  each  team  member.  The  form  should  include  each  question,  any  “listen  fors," 
document  requests,  and  room  for  notes. 


Approach 

G  Review  answers  to  the  questionnaire. 

□  Brainstorm  a  preliminary  set  of  questions. 

Q  Organize  and  eliminate  redundant  questions,  add  “listen  fors." 
G  Transcribe  questions  onto  forms  and  make  copies  for  team. 

G  Review  next  day’s  schedule. 
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Appendix  E:  SAM  Training  Support,  Continued 


Interview  SE  Leads 


Goals 


-  Resolve  any  misunderstandings  from  the  questionnaire. 

-  Clarify  any  anomalies  or  inconsistencies. 

-  Focus  assessment  team  on  process  areas  that  need  improvement. 


Approach 

The  team  leader  will  conduct  the  discussions.  All  other  team  members  should  take  notes.  Any 
team  member  can  ask  a  question,  although  the  team  leader  should  be  allowed  to  lead  the 
questioning  to  ensure  that  all  high-priority  questions  are  covered.  The  team  leader  should 
follow-up  on  the  “listen  fors"  and  note  any  document  requests. 

Q  Introduce  SE  lead  and  the  team. 

Q  Repeat  confidentiality  rules 

-  No  individuals  or  projects  named  in  results. 

-  They  may  not  disclose  comments  of  others  made  at  this  meeting. 

-  Team  will  take  notes.  All  notes  will  be  treated  as  confidential. 

Q  Explain  the  purpose  of  this  session. 

□  Cover  the  material  in  the  exploratory  questions. 

□  Remind  SE  lead  of  any  document  requests. 

Q  Remind  SE  lead  of  the  time  and  place  for  the  preliminary  findings  review. 

□  Thank  and  excuse  SE  lead. 
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Appendix  E:  SAM  Training  Support,  Continued 

Consolidate  Data  from  SE  Lead  Interviews 


Goals 


-  Give  team  members  a  chance  to  reflect  on  previous  sessions. 

-  Discuss  any  confusing  or  missing  information. 

-  Perform  adjustments  to  upcoming  activities. 


Approach 

These  sessions  give  the  team  a  chance  to  absorb  the  data  they  have  been  given  in  previous 
sessions.  They  begin  by  reviewing  their  notes  and  then  discuss  any  confusing  or  missing  data. 
Next,  changes  (if  any)  to  upcoming  activities  are  discussed. 

d  Quietly  review  notes. 

d  Discuss  issues. 

d  Adjust  upcoming  activities. 
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Appendix  E:  SAM  Training  Support,  continued 

Interview  Practitioners 


Goals 


-  Hear  concerns  and  views  from  the  perspective  of  the  practitioners. 

-  Verify  information  gathered  from  questionnaire  and  discussions  with  the  SE  leaders. 


-  Listen  for  new  process  issues  or  areas  of  concern. 


Approach 


Each  group  consists  of  6-10  professionals  considered  to  be  experts  and  opinion  leaders  in  the 
organization.  They  should  be  actually  working  on  projects  (i.e.,  not  staff  or  management). 
The  team  leader  opens  and  closes  each  session  and  facilitates  the  discussion.  Assessment  team 
members  need  to  relax  and  let  the  discussion  flow:  they  should  not  lead  the  discussions.  All 
assessment  team  members  should,  however,  take  notes  during  these  discussions.  Team 
members  should  occupy  every  other  seat  around  the  table  in  order  to  avoid  an  “us  vs.  them" 
atmosphere. 

LI  Introduce  the  topic  (if  appropriate). 

Li  Explain  the  conduct  of  the  meeting. 

-  How  the  operation  looks  to  them. 

-  Free-form  discussion  (team  will  not  ask  specific  questions). 

-  Chance  to  summarize  your  major  issues  at  the  end. 

Q  Repeat  confidentiality  rules. 

-  No  individuals  or  projects  named  in  results. 

-  They  may  not  disclose  comments  of  others  made  at  this  meeting. 

-  Team  will  take  notes.  All  notes  will  be  treated  as  confidential. 

Q  Introduce  everyone  (state  name  and  function). 

□  Turn  meeting  over  to  the  practitioners. 

LI  About  20-30  minutes  from  the  end  (5  minutes  per  person),  ask  each  practitioner 
If  you  could  change  one  thing  in  your  organization  other  then  your  boss  or  your 
paycheck,  what  would  it  be? 

Other  than  the  people,  what  do  you  think  is  this  organization’s  major  strength? 

G  Remind  practitioners  of  the  time  and  place  for  the  draft  briefing  presentation. 

□  Thank  and  excuse  the  practitioners. 
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Appendix  E:  SAM  Training  Support,  Continued 
Develop  Preliminary  Findings 


Goals 


—  Identify  list  of  key  process  issues. 


-  Generate  preliminary  findings. 


Approach 

Focus  on  issues  for  the  entire  organization.  It  is  important  that  the  findings  have  the  broadest 
possible  application,  both  to  preserve  confidentiality  and  have  maximum  impact  with  senior 
management.  Try  to  keep  the  goals  in  mind  and  use  the  team's  expertise  to  solve  the  problems. 
Avoid  the  following: 

—  Issues  without  useful  recommendations. 


-  Findings  based  on  hearsay. 

-  Sweeping  statements. 

Formulating  the  findings  is  the  most  difficult  part  of  the  assessment.  We  begin  with 
preliminary  findings  which  are  40-60  simple  statements.  At  this  point  in  the  assessment,  it  is 
not  necessary  to  have  team  consensus.  Nor  is  it  necessary  to  wordsmith  the  findings  although 
redundant  questions  should  be  consolidated. 

□  Brainstorm  a  list  of  findings  -  organized  by  PA. 

d  Review  each  PA  with  issues  and  eliminate  redundant  questions. 

Q  Transcribe  list  onto  form  and  make  copies  for  each  team  member. 
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Appendix  E:  SAM  Training  Support,  Continued 
Review  Preliminary  Findings 


Goals 

-  Get  feedback  on  preliminary  findings. 

-  Collect  any  requested  documentation. 

-  Ask  SE  leads  for  strengths  and  weakness. 


Approach 

The  team  leader  will  conduct  the  session,  and  all  other  team  members  should  take  notes.  Any 
team  member  can  ask  a  question,  although  the  team  leader  should  be  allowed  to  lead  the 
questioning  to  ensure  that  all  preliminary  findings  are  covered. 

d  Re-introduce  SE  lead  and  the  team. 

Q  Repeat  confidentiality  rules. 

-  No  individuals  or  projects  named  in  results. 

-  They  may  not  disclose  comments  of  others  made  at  this  meeting. 

-  Team  will  take  notes.  All  notes  will  be  treated  as  confidential. 

□  Collect  any  requested  documents. 

Q  Explain  the  purpose  of  this  session. 

□  State  each  preliminary  finding  and  ask: 

Is  the  finding  true  for  your  project? 

Is  the  finding  true  for  the  organization? 

□  About  five  minutes  from  the  end  ask: 

If  you  could  change  one  thing  in  your  organization  other  then  your  boss  or  your 
paycheck,  what  would  it  be? 

Other  then  the  people,  what  do  you  think  is  this  organization's  major  strength? 

□  Remind  SE  lead  of  the  time  and  place  for  the  draft  findings  presentation. 

O  Thank  and  excuse  SE  lead. 
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Appendix  E:  SAM  Training  Support,  Continued 

Develop  Draft  Rating 


Goals 


-  Establish  team  consensus  on  process  capability  profile. 


-  Produce  process  capability  profile  for  presentation  in  Final  Briefing. 


tiapante^;"  ^'^sessm^ntteam :  |  ,■  — 

.Durator  . . ^  . 


Approach 

As  a  result  of  the  data  gathering  that  has  taken  place  so  far,  team  members  should  have  updated 
DTSs  that  reflect  the  data  provided  via  the  questionnaires,  interviews,  and  any  documentation 
reviews  or  presentations  conducted.  The  profile  is  determined  prior  to  synthesizing  the  draft 
finds  so  as  to  inform  the  team's  decision  making  on  how  to  prioritize  the  findings.  The 
findings  should  provide  the  7-9  top  issues  that  improvement  should  focus  on — without  the 
rating  process  being  relatively  complete,  issues  which  may  have  voluble  supporters  but  do  not 
provide  significant  leverage  points  could  creep  into  the  findings.  The  rating  algorithm  for  SAM 
is  relatively  simple:  at  least  90%  of  applicable  base  practices  should  be  exhibited  across  the 
entire  sample  of  projects  selected  for  a  rating  of  "1"  to  be  achieved.  For  higher  levels,  100%  of 
lower  level  practices  (base  and  generic)  must  be  satisfied  and  90%  of  the  level's  GPs  must  be 
exhibited  across  the  projects  for  the  rating  of  that  level.  That  is,  if  there  are  4  GPs  and  3 
projects,  at  most  one  project  could  fail  to  perform  one  GP  to  still  achieve  the  rating  at  that  level 
(depending  on  the  capability  level)  of  the  generic  practices. 

d  Step  through  each  process  area. 

-  Review  notes  from  interviews 

-  Apply  rating  algorithm. 

-  Obtain  team  consensus  on  process  area  rating. 

Review  profile  as  a  whole  for  consistency. 

Q  Determine  presentation  style  for  profile  and  prepare  for  the  final  briefing. 
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Appendix  E:  SAM  Training  Support,  Continued 

Develop  Draft  Findings 


Goals 


-  Assign  a  level  of  practice  for  each  PA. 

-  Identify  list  of  (7  ±  2)  key  process  issues 


-  Generate  draft  findings  briefing. 


Approach 

The  assessment  team  will  assign  a  level  of  practice  (0-5)  for  each  PA.  Use  the  questionnaire 
analysis  and  notes  from  the  discussions  to  assist  in  this  process. 

Focus  on  issues  for  the  entire  organization.  It  is  important  that  the  findings  have  the  broadest 
possible  application,  both  to  preserve  confidentiality  and  have  maximum  impact  with  senior 
management.  Limit  the  number  of  findings  to  7  ±  2.  This  gives  definite  direction  for  process 
improvement  without  overloading  limited  resources.  Each  finding  should  consist  of 
Finding  -  A  single  statement  of  the  issue. 

Causes  -  Observations  that  contribute  to  the  finding. 

Consequences  -  Results  that  will  get  management  attention,  e.g.,  increased  rework. 

Preliminary  findings  are  often  found  to  be  causes  of  a  more  general  finding.  Both  findings  and 
causes  should  reflect  what  the  team  has  heard  from  the  SE  leads  and  the  practitioners.  The 
consequences  are  developed  by  the  team  to  ensure  management  attention,  and  need  not 
represent  consequences  voiced  by  the  SE  leader  or  the  practitioners.  Again,  avoid  issues 
without  useful  recommendations,  unsubstantiated  findings,  or  sweeping  generalizations. 

Formulating  findings  is  the  most  difficult  part  of  the  assessment.  It  is  the  place  where  the  team 
is  most  likely  to  have  conflict.  Try  to  keep  the  SE-CMM  in  mind  and  use  the  team’s  expertise 
to  solve  the  problems.  For  each  finding,  the  team  should 

Q  Identify  level  of  practice  for  each  PA. 

□  Review  notes  and  discussions  of  preliminary  findings. 

Q  Refine  issues  into  findings,  causes,  and  consequences. 

□  Form  mini-teams  to  wordsmith  individual  findings. 

Q  Review  each  finding  to  reach  consensus  on  wording  (entire  team). 

Q  Complete  briefing  and  produce  overheads  for  presentation. 
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Appendix  E:  SAM  Training  Support,  Continued 

Present  Draft  Findings 


Goals 

—  Provide  SE  leads  and  practitioners  with  the  opportunity  to  comment  on 
findings. 

-  Allow  the  team  to  judge  the  impact  of  the  findings  on  the  organization. 

-  Build  organizational  momentum  for  process  improvement. 


Approach 

The  assessment  team  leader  presents  the  findings  to  the  practitioners  and  SE  leads  in  separate 
sessions,  first  to  the  practitioners,  and  then  to  the  SE  leads.  These  groups  are  kept  separate  so 
opening  comments  are  not  inhibited.  The  ratings  are  not  presented  at  this  time.  In  each  session, 
the  findings  are  first  presented  without  interruption  so  that  the  audience  has  a  chance  to  hear  all 
of  the  findings.  The  presenter  should  use  the  exact  wording  from  the  slides.  The  presenter 
then  steps  through  each  finding  and  asks  for  comments.  The  assessment  team  members  may 
assist  the  leader  in  explaining  any  issues,  but  should  concentrate  on  taking  notes. 

□  Welcome  participants  and  set  the  context, 

d  Repeat  confidentiality  rules. 

-  No  individuals  or  projects  named  in  results. 

-  They  may  not  disclose  comments  of  others  made  at  this  meeting. 

-  Team  will  take  notes.  All  notes  will  be  treated  as  confidential. 

□  Make  presentation  (without  interruption). 

Repeat  each  finding  and  solicit  comments. 

Q  Remind  participants  of  the  time  and  place  of  the  final  presentation. 
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Appendix  E:  SAM  Training  Support,  Continued 

Adjust  Draft  Findings 


Goals 


-  Refine  wording  for  final  findings  presentation. 


-  Prepare  final  presentation. 


'PafWGipajtt^:"  ' 


. ''  -  i 


■iiii 


Approach 

As  a  result  of  hearing  the  findings  presentation  and  hearing  the  two  groups'  comments,  the 
team  will  see  places  where  the  focus  or  wording  of  some  of  the  findings  needs  refinement. 
The  goal  should  be  to  maximize  the  acceptance  of  the  assessment.  It  is  important  to  use  the 
assessment  to  build  momentum  for  process  improvement.  This  sometimes  requires  weakening 
or  strengthening  the  wording  for  a  finding. 

Therefore,  a  set  of  next  steps  should  be  scheduled  that  builds  on  the  findings  and  results  in 
observable  changes.  If  the  organization’s  expectations  are  not  satisfied,  the  opportunity  for 
change  may  be  lost  forever! 

Q  Step  through  each  finding. 

-  Review  notes  from  presentations. 

-  Refine  wording. 
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Appendix  E:  SAM  Training  Support,  continued 


Present  Final  Briefing 


Goals 


-  Visibly  present  the  results  of  the  assessment  to  senior  management. 

-  Build  support  for  addressing  the  findings. 

-  Review  next  steps. 


Approach 

The  assessment  team  leader  presents  the  final  findings  brief.  The  final  briefing  will  include 

1.  Assessment  scope  -  projects  &  participants  (thank  everyone!). 

2.  Rating  -  level  of  maturity  for  each  PA. 

3.  Strengths  -  organization's  strong  points. 

4.  Findings  -  adjusted  draft  findings. 

5.  Next  steps  -  findings  &  recommendations  report,  action  plan, . . . 


A-54 


SECMM-94-06ICMU/SEI-96-HB-004  v1 .1 


Appendix  E:  SAM  Training  Support,  continued 

Brief  Sponsor  (optional) 


Goals 


—  Provide  additional  background. 

-  Resolve  any  open  issues  with  senior  management. 


-  Discuss  next  steps. 


participants: 
Diiration: 


.  1  hour 


'V 


■ 


Approach 


This  optional  session  gives  senior  management  an  opportunity  to  ask  questions  and  discuss 
any  issues  that  they  were  reluctant  to  raise  in  the  open  form  of  the  final  briefing.  It  is  also  an 
opportunity  for  the  team  leader  (or  the  entire  team)  to  promote  follow-on  activities.  Remember 
that  the  confidentiality  rules  still  apply!  Do  not  let  senior  management  use  this  session  to  fix 
blame  for  any  problem  or  to  attribute  particular  findings  to  a  project  or  individual. 
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Appendix  E:  SAM  Training  Support,  Continued 


Conduct  Wrap-up 

Goals 

-  Evaluate  the  SE-CMM. 

-  Evaluate  the  assessment  process. 

-  Plan  next  steps. 

Approach 

The  assessment  team  uses  this  session  to  generate  feedback  on  the  SE-CMM  pilot  assessment. 
Each  team  member  completes  the  questionnaire  evaluation  form.  These  results,  along  with  the 
questionnaire  evaluation  form  from  the  SE  leads  is  returned  to  the  SE-CMM  authors  for 
review.  Next,  a  discussion  focuses  on  the  model  and  the  assessment  process. 

Q  Each  team  member  completes  a  questionnaire  evaluation  form. 

G  The  team  discusses  the  SE-CMM.  What  works?  What  doesn't?  What’s  missing? 

G  The  team  discusses  what  in  the  assessment  process  worked  and  what  did  not. 

Before  the  team  breaks,  the  next  steps  should  be  scheduled  and  responsibilities  assigned. 

G  Discuss  next  steps. 

-  Findings  and  recommendations  report? 

-  Action  plan? 

G  Schedule  next  steps. 
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Appendix  E:  SAM  Training  Support,  continued 
Post-On-Site  Week  Activities 


Goals 

-  Document  findings  for  future  comparison 

-  Develop,  validate,  and  present  recommendations  to  mitigate  findings 

-  Establish  sponsorship  for  improvement  efforts 

-  Develop  action  plans  to  implement  recommendations 

-  Perform  action  plans 


Approach 

This  phase  is  where  the  organization’s  wheels  meet  the  road.  The  findings  must  be 
documented  and  this  is  usually  done  with  the  recommendations  as  a  short  report.  The  report 
may  be  separate  from,  or  included  as  an  appendix  to,  the  Action  Plans  developed  to  implement 
or  deploy  the  recommendations.  Part  of  the  Action  Plans  is  the  priorities  and  schedule  for 
implementation.  Often  an  organization  needs  to  prototype  its  change  mechanisms  on  one 
project  before  trying  to  extend  them  to  the  organization  as  a  whole,  or  place  infrastructure  such 
as  a  software,  systems,  or  product  process  group  in  place  with  resources  and  change  charter. 
Sponsorship  for  the  appraisal  must  also  be  extended  to  the  implementation  of  changes. 
Developing  this  sponsorship  is  non-trivial.  Finally,  the  action  plans  must  be  turned  on  and 
allowed  to  run  before  a  re-appraisal  can  show  that  improvements  have  happened  (or  not)  and 
permit  a  refocus  of  organizational  change  efforts. 
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Appendix  E:  SAM  Training  Support,  continued 

A  Technique  for  Recommendation  Generation  and  Documentation 
of  Findings  and  Recommendations 


•  Identify  selected  personnel  for  their  knowledge  in  the  subject  area  of  a  finding 
(especially  those  who  seemed  to  be  doing  the  area  well  or  better  than  others) 

•  With  the  appraisal  team  or  a  subset,  review  the  finding  and  develop  a  common  vision  of 
the  root  causes  if  not  evident  in  the  findings  statement  or  supporting  findings 
(causes/consequences). 

•  Perform  a  solution  generation  tree  diagram  exercise  (Ref:  “Memory  Jogger  Plus+” 
Program  Decision  Process  Chart  [PDPC  technique]  description) 

-  Write  the  problem  statement  (what  is  to  be  changed  from  the  finding) 

-  Brainstorm  possible  solutions/actions  that  would  eliminate  or  reduce  its  impact 

-  For  each  alternative,  brainstorm  the  consequences  (costs,  benefits,  other  side  effects) 

-  For  each  consequence,  come  to  consensus  and  record  whether  it  is  +,  -,  or 
implementation-dependent  on  whether  it  would  be  good  (-I-)  or  bad  (-)  and  leave 
blank 

-  Collect  those  alternatives  with  mostly  +  or  blank  consequences  -  these  are  low-risk 
actions  to  eliminate  or  mitigate  the  problem! 

-  Write  these  up  as  a  recommendation  in  format:  Finding,  Recommendation,  Rationale 
(no  more  than  1-3  paragraphs) 

•  Brief  the  practitioners  group  and  ask  for  improvement  suggestions 

•  Incorporate  as  the  team  feels  need 

•  Brief  the  systems  engineering  or  project  leaders  (interviewees)  and  ask  for 
improvement  suggestions  and  any  political  sensitivities  that  might  require  better 
wording 

•  Incorporate  as  indicated 

•  Deliver  the  report  and  brief  the  sponsor  within  a  week  for  feedback.  At  the  meeting 
work  on  what  will  be  needed  for  long  term  sponsorship  of  the  change  effort,  and  the 
desired  format  for  action  plans  if  any  (offer  that  below  or  the  organization’s  “usual” 
plan  template) 
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Appendix  E:  SAM  Training  Support,  Continued 


A  Technique  for  Action  Planning 

•  Identify  and  obtain  participation  of  those  with  knowledge,  skills,  and  organizational 
influence  useful  to  plan  success  and  efficiency  (one  organization  requires  participation 
in  this  of  at  least  one  “Archie  Bunker”  to  ensure  broad  enough  understanding  of 
resistance  to  change) 

•  Meet  on  each  recommendation  having  prepared  by  reviewing  the  entire  set  (for  leverage 
between  them)  and  the  preferred  format  or  template 

-  Develop  measures  of  success  (observable  outcomes  that  can  serve  as  exit  criteria  so 
that  an  objective  outsider  could  say  when  the  project  was  done) 

-  Develop  output  descriptions  (what  will  the  outputs  needed  to  achieve  the  success 
criteria  look  like  or  be  like?  Make  these  as  measurable  as  possible.) 

-  Develop  validation/verification  mechanisms  (who  will  check  what  and  when  during 
and  at  the  end  the  of  the  plan  to  ensure  it’s  working  the  way  it  was  planned  and 
achieving  the  desired  outcomes?) 

-  Develop  entry  criteria  -  what  must  be  done  before  the  action  plan  can  begin 

-  Develop  inputs  descriptions  -  what  will  be  needed  from  others  during  the  plan 

-  Develop  task  descriptions  (high  level  or  detailed  flow  charts  or  deployment  charts  are 
fine) 

-  Much  of  this  can  be  developed  by  assignment  to  subgroups,  but  then  the  action 
planning  team  must  peer  review  the  sections  to  ensure  everyone  agrees  that  it  is  not 
too  much  or  too  little,  and  that  it  provides  sufficient  motivation  and  achievement  to 
do 

-  After  the  initi^  proposal,  a  force  field  analysis  (Ref:  “Memory  Jogger”)  should  be 
performed  to  identify  risks  and  leverage  forces  and  develop  contingencies  if  needed 

•  Write  the  action  plan  title  at  the  top  of  a  flip  chart 

•  Draw  a  vertical  line  down  the  page  under  the  title 

•  On  the  left  write  “forces  for”,  and  on  the  right,  “barriers  to” 

•  Brainstorm  both  and  record  on  the  flip  chart(s) 

•  Assign  consensus  or  median  weights  from  1  (trivial  force  for  or  barrier  to  the  plan 
working)  to  5  (overwhelming  force  for,  or  insurmountable  barrier  to  the  plan 
working) 

•  Add  the  weights  and  if  the  ratio  is  3:2  or  better  for  the  plan,  write  it  up  and  submit 
it 

•  If  the  ratio  is  less  than  2:3  for,  then  consider  seriously  NOT  suggesting  the  plan 

•  If  the  ratio  is  between  2:3  and  3:2,  draw  a  line  at  the  bottom  of  the  flips  and  on  the 
barriers  side,  write  “Additional  costs”  with  a  force  of  4.  Brainstorm  things  that 
could  be  done  for  more  resources  or  sponsorship,  etc.  that  could  overcome  each 
barrier  of  level  3  or  4  (or  5),  until  the  total  of  the  original  force  field  ratio  and  the 
additional  4  barrier  and  contingencies  on  the  forces  for  side  surpass  3:2  ratio. 
The  Action  Plan  write  up  then  includes  these  extra  forces  that  cost  more  than  your 
original  plan  as  “contingencies”  to  be  executed  if  the  barriers  become  real  -  that  is, 
they  are  risk  mitigations. 


A-62 


SECMM-94-06ICMU/SEI-96-HB-004  v1 .1 


Appendix  E:  SAM  Training  Support,  Continued 


-  Prioritize  and  consider  the  sequencing  of  the  action  plans  for  each  finding.  Which 
ones  are  least  risky  but  provide  more  than  nominal  impact  or  improvement?  Make 
these  first.  Ensure  you  pilot  actions  on  a  “friendly  group”  before  deploying  to  the 
whole  organization.  Note  that  the  schedule  will  be  longer  than  anyone  hopes  for  or 
expects  routinely.  Consider  how  to  shorten  it,  but  stick  to  your  estimates  the  first 
time. 
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Appendix  F:  Site  Coordination  Checklist 


Introduction 


Preparation  tasks 
(prior  to  on-site 
period) 


This  checklist  is  used  to  support  the  SE-CMM  appraisal  site  coordinator 
in  preparing  for  the  on-site  period  of  the  SE-CMM  appraisal  method 
(SAM). 


Major  events  that  the  site  coordinator  is  responsible  for  arranging  are 
described  in  the  table  below.  The  time  frames  given  are  approximate, 
and  are  based  around  the  beginning  of  the  on-site  period.  The 
coordinator  can  use  this  list  as  a  checklist  for  preparation. 


Task 

Description 

Time  frame 

Executive 
briefing  to  obtain 
sponsorship  for 
SAM 

Briefing  by  site  coordinator, 
appraisS  team  leader  or 
facilitator,  as  appropriate,  to  the 
potential  sponsor  introducing 
the  SE-CMM  and  SAM 
concepts. 

At  least  six 
weeks  prior 
to  on-site 
period 

Determine 

confidentiality 

requirements 

Determine  with  senior 
management  the  need  for 
nondisclosure  and 
confidentiality  of  agreements. 

Four  to  six 
weeks  prior 
to  on-site 
period 

Select  projects 
(three  to  four 
projects) 

Site  coordinator,  working  with 
the  sponsoring  manager,  selects 
projects  appropriate  to  the 
appraisal  purpose. 

Four  to  six 
weeks  prior 
to  on-site 
period 

Determine  on¬ 
site  week 

Working  with  senior 
management  and  the  facilitator, 
determine  a  week  when  the 
systems  engineering  leads, 
senior  management,  facilitators, 
and  potential  appraisal  team 
members  are  available  as 
needed. 

Four  to  six 
weeks  prior 
to  on-site 
period 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator. 


continued  on  next  page 
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Appendix  F:  Site  Coordinator  Checklist,  continued 


Preparation 
tasks,  continued 


~r 

Task 

Description 

Time  frame 

Select 
appraisal 
team  (five  to 
seven  team 
members 
plus  one  to 
two 

facilitators) 

Site  coordinator  typically 
determines  the  pool  of 
appraisal  team  members  for 
consideration  by 
management,  unless  the 
coordinator  has  been 
empowered  to  make  the 
selection,  in  which  case  the 
selections  are  made.  The 
facilitator  is  usually  available 
to  help  in  screening 
candidates.  Provide  a  copy 
of  the  SE-CMM  and  SAM 
description  to  the  appraisal 
team  members. 

Four  to  six 
weeks  prior  to 
on-site  period; 
goal  is  to  select 
the  team  far 
enough  in 
advance  to  be 
able  to  get  the 
team  members 
to  schedule  the 
appraisal  week 

Select 

systems 

engineering 

lead 

For  each  project  selected, 
identify  the  systems 
engineering  leads  for  the 
selected  projects  and  talk  to 
them  about  their 
involvement.  Also,  verify 
their  availability  during  the 
appraisal  week 
(approximately  seven  hours 
per  person). 

Three  to  four 
weeks  prior  to 
on-site  period 

- 

Select 

practitioners 

Select  potential  interviewees 
in  the  selected  projects  as 
well  as  other  projects  in  the 
organization  being  appraised. 
Typically,  the  facilitator  is 
available  to  consult  on 
participants,  and  management 
typically  approves  the 
participant  list. 

Three  to  four 
weeks  prior  to 
on-site  period 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator,  continued 


continued  on  next  page 
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Appendix  F:  Site  Coordinator  Checklist,  continued 


Preparation 
tasks,  continued 


Task 

Description 

Time  frame 

Administer 

questionnaire 

The  selected  systems 
engineering  leads  (and  other 
practitioners,  if  selected  for 
questionnaire  analysis)  are 
provided  with  the  SAM 
questionnaire,  and  the  site 
coordinator  provides  the  time 
frame  for  returning  the 
questionnaire.  Hand  out  the 
SE-CMM  glossary  with  the 
questionnaire.  The  site 
coordinator  should  be 
available  to  answer  any 
clarifying  questions;  usually 
a  one  or  two  day  turnaround 
is  requested  if  the 
questionnaires  are  not 
completed. 

Two  weeks 
prior  to  on-site 
period 

Handout 

about 

questionnaire 

For  pilots,  when  the 
questionnaire  is  handed  out, 
also  hand  out  the 
questionnaire  about  the 
questionnaire  and  emphasize 
the  need  for  feedback  from 
people  who  answer  the 
questionnaire. 

Two  weeks 
prior  to  on-site 
period 

Collect 

questionnaire 

The  site  coordinator  collects 
the  completed  questionnaires, 
makes  a  copy  for  disaster 
recovery  purposes,  and  mails 
questionnaires  back  to  the 
facilitator  for  initial  analysis. 

To  be  received 
by  facilitator 
one  week  prior 
to  on-site 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator,  continued 


continued  on  next  page 
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Appendix  F:  Site  Coordinator  Checklist,  continued 


Preparation 
tasks,  continued 


~1~~ 

Task 

Description 

Time  frame 

Prepare 

appraisal 

team 

notebook 

Prepare  a  three-hole  binder 
(for  each  appraisal  team 
member)  that  contains  copies 
of  the  completed 
questionnaires  and  blank 
paper  for  taking  notes.  Tabs 
for  each  project,  each 
practitioner  discussion, 
preliminary  findings,  and 
findings  briefing  are  often 
helpful.  These  notebooks  are 
handed  out  during  the  team 
training  at  the  beginning  of 
the  on-site  period. 

One  week  prior 
to  on-site  period 

Schedule 
rooms  for 
opening  and 
final  briefing 
rooms 

Schedule  rooms  large  enough 
to  hold  all  anticipated 
participants,  including 
management,  appraisal 
participants,  and  other 
identified  invitees. 

Two  weeks 
prior  to  on-site 
period, 
depending  on 
how  tight 
facilities  are 

Schedule 
rooms  for 
practitioner 
interviews 

Schedule  rooms  large  enough 
to  hold  20  people  for  the 
practitioner  interviews. 

Two  weeks 
prior  to  on-site 
period, 
depending  on 
how  tight 
facilities  are 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator,  continued 


continued  on  next  page 
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Appendix  F:  Site  Coordinator  Checklist,  Continued 


Preparation 
tasks,  continued 


Task 

Description 

Time  frame 

Schedule 
rooms  for 
systems 
engineering 
lead 

interviews 

and 

assessment 

team 

Schedule  a  room  large 
enough  for  10  people  plus 
some  extra  workspace  that 
can  be  blocked  off  for  team 
use  for  the  entire  week  of  the 
on-site  period.  Preferably, 
this  area  is  somewhere  that 
can  be  locked  at  night,  but 
not  in  an  area  that  will  require 
escort  of  the  appraisal  team 
members/facilitators  during 
the  week. 

Two  to  three 
weeks  prior  to 
on-site  period, 
depending  on 
how  tight 
facilities  are 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator,  continued 
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Appendix  F:  Site  Coordinator  Checklist,  continued 


Preparation 
tasks,  continued 


Task 

Description 

Time  frame 

Schedule 

support 

facilities 

The  following  support  tools 

should  be  provided: 

•  If  possible,  a  personal 
computer  of  the  type  most 
often  used  in  the 
organization  (usually 
Windows-based  or  Mac) 
and  an  associated  dedicated 
laser  printer  scheduled  for 
the  week.  (Desired 
software  is  MS  Word  and 
MS  PowerPoint  for  current 
facilitator  group.) 

•  Instructions  for  obtaining 
photocopies  in  the  building 
being  used.  (This  is  not 
much  of  an  issue  if  a 
building  familiar  to  the 
appraisal  team  members  is 
being  used.) 

•  Lists  of  facilities,  hotels, 
and  restaurants  that  are 
close  by  (preferably  with  a 
map)  and  information  on 
restaurant  delivery  service. 

•  Flip  charts/markers, 
transparencies,  notepads, 
power  strips,  3-hole 
punch,  binders,  lots  of 
post-it  notes. 

Computer  may 
take  several 
weeks  to 
arrange;  other 
arrangements 
just  need  to  be 
made  prior  to 
on-site  period 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator,  continued 
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Appendix  F:  Site  Coordinator  Checklist,  continued 


Preparation 
tasks,  continued 


Task 

Description 

Time  frame 

Schedule 
support  staff 

If  possible,  provide  dedicated 
secretarial  support  for  the  last 
three  days  of  the  on-site 
period  to  provide 
transcription,  revision, 
printing,  reproduction  and 
note-taking  services. 

(In  many  cases  the  appraisal 
team  ends  up  doing  their  own 
support,  but  having  a 
secretary  to  do  these  tasks 
can  really  relieve  some  of  the 
grunge  work.) 

Two  to  three 
weeks  prior  to 
on-site  period 

Verify  senior 

management 

schedule 

Verify  that  senior 
management  is  prepared  to 
attend  and  speak  at  the 
opening  briefing,  and  attend 
the  fin^  briefing. 

Two  weeks, 
then  one  week 
prior  to  on-site 
period 

Verify 

participant 

schedule 

Verify  that  participants  are 
available  in  their  Plotted  time 
slots;  make  any  revisions 
necessary  to  the  schedule. 

Two  weeks, 
then  one  week 
prior  to  on-site 
period 

Verify  team 

member 

schedule 

Verify  that  appraisal  team 
members  have  no  conflicts 
during  the  on-site  period. 

Two  weeks, 
then  one  week 
prior  to  on-site 
period 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator,  continued 


A-70 


SECMM-94-06ICMU/SEI-96-HB-004  v1 .1 


Appendix  G:  Approved  SAM  Requirements 


Introduction 


1.0  Scope 


2.0  Applicability 


3.0  Pre-on-site 
work 


This  appendix  contains  the  requirements  for  SAM  approved  by  the  SE- 
CMM  Steering  Group. 

The  following  requirements  are  synthesized  from  the  "Sources  of 
Requirements"  v.3  used  at  the  3/8/94  SE-CMM  author's  meeting.  In 
addition,  some  requirements  (i.e.,  those  in  6.0  and  7.0)  are  derived 
from  the  SEI's  Common  Appraisal  Framework  (CAF),  with  which  the 
method  is  intended  to  be  compatible.  Ultimate  compatibility  with  the 
CAF  depends  on  the  final  form  of  that  framework. 


1 . 1  The  1 994  SE-CMM  Appraisal  Method  (SAM)  scoj^  is  limited  to 
assessment  of  the  appraised  entity's  process  capability. 

1.2  The  SAM  deals  with  the  diagnostic  phase  of  a  process  improvement 
program. 

1.3  vl.O  of  the  SAM  is  focused  on  appraisal  to  support  self- 
improvement. 


The  SAM  will  be 

2.1  Applicable  to  multiple  types  of  appraised  entities  (e.g.,  sites, 
organizations,  and  projects). 

2.2  Adaptable  to  in-house  process  improvement  situations. 

2.3  Adaptable  to  supplier  selection  situations. 

2.4  Applicable  to  contract-driven  environments. 

2.5  Applicable  to  market-driven  environments 


3. 1  Pre-on-site  work  for  participants  of  SAM  will  be  limited  to  four 
hours  per  appraisal  participant. 

3.2  Pre-on-site  work  for  SAM  appraisal  team  members  will  be  limited 
to  40  hours  per  appraisal  team  member,  not  including  training. 

3.3  SAM  appraisal  team  members  will  be  trained  prior  to  participation 
in  an  SE-CMM  appraisal. 


continued  on  next  page 
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Appendix  G:  Approved  SAM  Requirements,  Continued 


4.0  On-site  work 

4.1  On-site  work  for  appraisal  participants  will  be  limited  to  five 
calendar  days,  of  which  not  more  than  two  staff-days  per  person 
will  be  required  over  that  time. 

4.2  SAM  will  target  8-10  hour  days  for  skilled  appraisal  team 
members. 


5.0  Post  on-site  5.1  Post-on-site  work  for  appraisal  participants  will  be  limited  to  four 
work  hours  per  participant. 

5.2  Post-on-site  work  for  appraisal  team  members  to  complete  the 
SAM-related  work  will  be  limited  to  40  hours  per  team  member. 

continued  on  next  page 
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Appendix  G:  Approved  SAM  Requirements,  continued 


6.0  CAF 
conformance 


Note:  The  SE-CMM  project  intends  for  the  SAM  to  conform  to  the 

SEI's  common  appraisal  framework.  This  satisfies  a  higher  level 

project  requirement  that  the  SE-CMM  avoid  conflicts  with  the  CMM. 

The  individual  requirements  below  collectively  serve  this  purpose. 

6. 1  The  SE-CMM  is  the  reference  model  for  the  SAM. 

6.2  No  applicable  part  of  the  SE-CMM  is  excluded  in  the  SAM. 

6.3  SAM  uses  the  CAF  rating  scale: 

Satisfied 
Not  satisfied 
Not  applicable 
Not  rated 

6.4  Judgments  made  as  part  of  SAM  are  made  by  the  appraisal  team. 

6.5  Judgments  made  by  the  appraisal  team  address  base  practices, 
generic  practices,  and  process  areas. 

6.6  CAF  rules  of  evidence  will  be  applied  in  SAM;  i.e.,  data  will  be 
corroborated  by  multiple  sources.  (See  CAF  for  specific  rules  of 
evidence.) 

6.7  Confidence  ratings  will  be  determined  as  defined  in  the  CAF,  i.e., 
confidence  associated  with  both  the  criteria  selected  and  the 
execution  of  the  method  will  be  addressed. 

6.8  The  SAM  will  document  how  and  where  it  conforms  with  the  CAF. 

6.9  The  SAM  will  limit  the  number  of  appraisal  team  members  to  a 
minimum  of  four  and  a  maximum  of  eight. 

6.10  The  SAM  will  require  the  appraisal  team  to  document  the  domain 
of  the  appraisal  (project,  organization,  site). 
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Appendix  G:  Approved  SAM  Requirements,  Continued 


7.0  Appraisal 
activities 

The  following  activities  are  required  to  be  addressed  in  the  SAM,  to 
achieve  CAP  conformance: 

7.1  Planning 

7.2  Selection 

7.2. 1  Selection  of  appraised  entity 

7.2.2  Selection  of  appraisal  team 

7.3  Data  collection 

7.4  Data  consolidation 

7.5  Rating 

7.6  Reporting 

7.6  Post-appraisal  activities 

8.0  Confidence 

8. 1  SAM  will  address  issues  related  to  confidence  and  risk  in  versions 
beyond  v  1 .0.  Version  1 .0  of  SAM  does  not  meet  the  CAF  requirement 
to  address  confidence  and  risk. 

9.0  Tailoring 

9. 1  SAM  will  describe  limits  of  tailoring  expected. 

10.0  Coverage 

10.1  SAM  will  describe  coverage  requirements  related  to  a  particular 
confidence  rating. 

11.0  Support 
materials 

SAM  will  describe 

11.1  The  training  materials  required  for  an  SE-CMM  appraisal. 

1 1 .2  The  supporting  briefing  materials  required  for  an  SE-CMM 
appraisal. 


1 1 .3  The  supporting  data-gathering  materials  required  for  an  SE-CMM 

appraisal. 

1 1.4  Data-analysis  materials  required  for  an  SE-CMM  appraisal. 
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Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF) 


The  CAF  is  a  standard  against  which  appraisal  methods  can  be 
compared.  Compliance  generally  means  that  the  appraisal  method  will 
provide  an  accurate,  repeatable  set  of  findings  and  rating  according  to 
the  model,  if  used  in  its  domain  of  reference. 


SAM/CAF 

conformance 

matrix 


Requirement  in  CAF  1.0 

SAM  Paragraph 

Rl.  Method  documentation 
references: 

•  Reference  to  CMM  version 

Abstract 

1.1.2,  Assumptions 

•  Reference  to  CAF  version 

This  Appendix 

•  How  it  implements  CAF  appraisal 
activities 

1.1.1  Phases 

1.1.1  Relate  to  CB  A-IPI 

•  How  it  implements  CAF  appraisal 
artifacts 

1.1.3  Plan  Appraisal  Details 

1.1.4  Exit  Criteria 

2.2.5  Consolidate  Data  1 

2.2.6  Interview  Practitioners 

2.2.7  Consolidate  Data  2 

2.2.9  Review  Prelim  Findings 

2.2. 10  Develop  Draft  Rating 

2.2.13  Adjust  Draft  Rating 

2.2.14  Final  Briefing 

2.2.15  Sponsor  Briefing 

2.2.16  Appraisal  Wrap-up 

•  How  it  implements  CAF  appraisal 
guidance 

This  Appendix 

R2.  Guidance  on: 

•  Planning  and  preparing  for 
appraisal 

2.1  Preparation 

•  Conducting  appraisal 

2.1.4  to  2.3.3 

•  Reporting  results 

2.2.13  to  2.2.14  and 

2.3.1  to  2.3.2 

Table  H-1.  Traceability  Matrix  to  SEI  CAF 
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Appendix  H:  Traceability  Matrix  to  SEi  CMM  Appraisal 
Framework  (CAF),  Continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

R3.  Guidance  for: 

•  Identifying  appraisal  goals 

•  Identifying  appraisal  constraints 

•  Determining  suitability  wrt 
goals/constraints/Abstract  to 

Scope 

2.1.1  Sponsor  Commit 

2.1.1  Sponsor  Commit 

2.1.2  Appraisal  Parameters 

2.1.3  Appraisal.  Details 

2.1.1  Sponsor  Commit 

R4.  Guidance  to  select  CMM  scope 

2.1.1  Purpose/Summary 
*2.1.1  Tailor  Parameters 

R5.  Guidance  to  select  organizational 
scope 

2.1.2  Summary  Description 

R6.  Guidance  to  obtain  organization, 
commitment 

2.1.1  to  2.1.3 

R7.  Appraisal  team  qualification 

criteria: 

•  >=  5  yrs  experience  for  majority 
of  team 

•  >=  25  yrs  experience  total  on  team 

•  >=  6  yrs  management  experience 
for  manager  on  team 

•  >=10  yrs  management  experience 
total  for  team 

*  1.3  Roles 

*  1.3  Roles 

*  1.3  Roles 

*  1.3  Roles 

R8.  Appraisal  team  leader  has 
experience:  ** 

•  Using  appraisal  method 

•  Managing  teams 

•  Facilitating  group  discussions 

•  Making  presentations 

*  1.1.3  Facilitator 

*  1.1.3  Roles 

1.1.3  Facilitator 

*  1.1.3  AT  leader 

R9.  Guidance  for  determining 
appropriate  team  size 

1.1.3  (1st  paragraph.) 

2.1.2  Table  2-5 
*  target  =  6  members 

Table  H-1.  Traceability  Matrix  to  SEI  CAF,  continued 


continued  on  next  page 


A-76 


SECMM-94-06ICMU/SEI-96-HB-004  v1 .1 


Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

RIO.  Guidance  on  preparing  a  team  to 
do  appraisal 

1.4  Collect  data 

2.2.1  Table  2-10 

2.2.2  SAM  training 

R1 1.  Guidance  for  site  selection 

*  2.1.2  Select  Parameters 

*  based  on  goals/project  status 

R12.  Guidance  for  project  selection 

*  2.1.2  Select  Parameters 

*  based  on  goals/project  status 

R13.  Guidance  for  participant 
selection 

1.1.3  Roles 

1.1.4  to  2.1.4 

R14.  Guidance  for  appraisal 
participants 

2.2.1 

R15.  Guidance  for  appraisal  planning: 

•  Identifies  appraisal  goals 

•  Identifies  appraisal  scope 

•  Identifies  appraisal  activities 

•  Provides  appraisal  schedule 

•  Identifies  resources 

•  Identifies  outputs  and  their  usage 

•  Identifies  anticipated  follow-on 
activities 

•  Documents  tailoring  and  trade-offs 

•  Identifies  risks  with  appraisal 
execution 

2.1.1  Sponsorship 

2.1.1  to  2.1.3 

2.1.3  Plan 

2.0  On-site 
(missing  figure) 

2.1.2  Summary 

2.1.3  Summary 

2.1.3  Summary 

2.3.0  Post- Appraisal 

*  2.3.4  Develop  Report 

1.2  Summary/Tailor 

*  2.1.3  Summary 

*  plan  includes  "force  field" 

R16.  Guidance  for  time  to  conduct 
appraisal 

2.1.0  Typical  Duration 

R17.  Guidance  for  appraisal  logistics 

2.1.3  Summary 
*  hours,  meals,  space,  etc. 
Appendix  C 

Table  H-1.  Traceability  Matrix  to  SEI  CAF,  continued 
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Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  Continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

R18.  Define  artifacts  for: 

•  Recording  observations 

2. 1 .4  Questionnaire 

•  Categorizing  observations 

2.2.3  Generate  Qs 

Appendix  C 

2.2.5  Data  Tracking  Sheet 

•  Classifying  observations 

2.2.3  Summary 

•  Validating  observations 

2.2.5  Update  DTS 

•  Recording  coverage 

2.2.6  Summary 

2.2.7  Update  DTS 

2.2.9  Review  Finds 

2.2.10  Update  DTS 

*  Use  PA  graphics  with 

•  Making  rating  decisions 

findings  stickies  to  display 
coverage 

2.2.7  Prelim  ratings 

2.2. 10  Draft  rating 

Reporting  findings  and  ratings 

2.2.12  Present  Finds 

2.2.14  Present  brief 

Managing  logistics 

2.2.15  Brief  sponsor 

2.3.2  Output  to  others 

2.1.3  Notes 

Appendix  G,  coord  checklist 

Table  H-1.  Traceability  Matrix 

to  SEI  CAF,  continued 
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Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

R19.  Guidance  to  implement  data 
collection  techniques: 

•  Administering  instruments 

2.1.4  Appraisal  participants 

•  Conducting  presentations 

2.2.1  Table  2-10 

2.2.9  Table  2-26 

2.2.12  Table  2-32 
*2.2.14  Table  2-36 
*2.2.15  Table  2-38 

•  Conducting  interviews 

2.2.4  Table  2-16 

2.2.6  Table  2-20 

•  Reviewing  documentation 

*  2.2.3  Table  2-14 

*  suggestions  for  process 
documentation/artifacts  to  ask 

for  and  quality  attributes  (add 
appendix  template) 

R20.  Guidance  for  collecting  data: 

•  Extracting  data  from  data 
gathering  sessions 

•  Recording  data  as  observations 

2.2.3  Table  2-14 
*  needs  explicit  map  of 
instmment  answers  to  KPAs 
2.2.5  Table  2-18 

2.2.7  Table  2-22 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

2.2.5  Table  2-18 

2.2.7  Table  2-22 

2.2.10  Table  2-28 

•  Classifying  observations 

•  Categorizing  as  CMM/non-CMM 

*  2.2.3  Table  2-14 

findings 

*  note  how  to  set  up  KPA 
graphics  with  non-CMM 
section(s) 

•  Categorizing  as  CMM/non-CMM 

*  2.2.8  Table  2-24 

findings 

*2.2.11  Table  2-30 

Table  H-1.  Traceability  Matrix 

to  SEi  CAF,  continued 
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Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

R21.  Guidance  for  validating 
observations: 

•  Corroboration  from  multiple, 
independent  sources 

•  Interviews  by  doers  of  work  or 
outcome  document  reviews 

2.2.5  Table  2-18 

2.2.7  Table  2-22 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

2.2.6  interviews 

R22.  Guidance  to  validate  a  portion  of 
interview  data  by  KPA  (goal-related) 
documentation 

2.1.4  Notes 

2.2.3  Notes 

2.2.4  Notes 

2.2.6  Notes 

R23.  Guidance  for  observation 
coverage  of  scope  and 
institutionalization: 

•  Each  goal  satisfied 

•  Each  KPA  institutionalized 
(common  features) 

•  Each  ML  is  satisfied 

Not  applicable 

*  2.1.4  Summary 

*  How  generic  practices  are 
covered  in  Questionnaire  and 
document  review 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

Not  applicable 

R24.  Mechanisms  to  adjust  collection 
to  obtain  coverage 

2.2.3  Table  2-14 

2.2.5  Table  2-18 

2.2.7  Table  2-22 

2.2.10  Table  2-28 

R25.  Guidance  for  collected  data 
traceability  to  outputs 

*2.1.4  Summary 
*  Add  graphic  display  of  +!- 
data  from  Questionnaire 

2.2.5  Table  2-18 

2.2.7  Table  2-22 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

Table  H-1.  Traceability  Matrix  to  SEI  CAF,  continued 
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Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  Continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

R26.  Require  ratings  of: 

•  KPA: 

•  Goals 

*  2.1.4  Summary 

*  How  generic  practices  are 
covered  in  Questionnaire  and 
document  review 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

Not  applicable 

R27.  If  appraisal  calculates  maturity 
level  rating,  consistent  w/  five  level 
scale  in  CMM  for  SW 

Not  applicable 

R28.  Rating  process  uses  the  rating 
values: 

•  Satisfied 

•  Unsatisfied 

•  Not  applicable 

•  Not  rated 

Not  applicable 

R29.  Rating  process  specifies: 

•  Goals  can  be  rated  when  coverage 
is  sufficient 

•  KPAs  can  be  rated  when  goals 
have  been 

•  Maturity  level  can  be  rated  when 
KPAs  have  been 

Not  applicable  (goals) 

Not  applicable  (goals) 

Not  applicable  (ML) 

R30.  Rating  process  uses  consensus 
of  team 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

R3 1 .  Ratings  are  based  on  CMM  for 
software,  Vl.l 

Not  applicable 

R32.  Rating  process  requires  ratings 
to  be  based  on  findings 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

R33.  Rating  process  specifies  goals 
are  rated: 

•  Satisfied  (conditions) 

•  Unsatisfied  (conditions) 

•  Not  applicable  (conditions) 

•  Not  rated  (conditions) 

Not  Applicable  (there  are  no 
goals  included  in  the  SE-CMM 
model) 

Table  H-1.  Traceability  Matrix  to  SEI  CAF,  continued 


continued  on  next  page 


SECMM-94-06ICMU/SEI-96-HB-004  v1 .1 


A-81 


Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

R34.  Rating  process  for  KPAs  based 
on  goals 

Not  applicable 

R35.  Rating  process  which  specifies 
maturity  level  by  KPA  satisfaction 

Not  Applicable 

R36.  Reports  the  team  provides  are 
identified: 

2.1.1  Tailor  Parameters 

2.2.14  Summary 

2.3.1  Lessons  learned 

2.3.2  Report  to  others 

R37.  Reporting  includes  the  following 
data: 

•  Scope 

•  Selections  (site,  projects, 
participants,  team) 

•  Findings 

•  Ratings 

•  Risks  associated  with 
accuracy/completeness 

Appendix  B  (example) 

Appendix  B  (example) 

2.2.11  Table  2-30 

2.2.13  Table  2-34 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

*  22.2.10  Table  2-28 

*  Appendix  B  (example) 

R38.  Guidance  for  protecting 
confidentiality 

Appendix  (^estionnaire 

2.2.1  Table  2-10 

Appendix  F  Training  Materials 
2.2.4  Table  2-16 

2.2.6  Table  2-20 

*  2.2.9  Notes 

*  add  note  to  remind  of 
confidentiality 

2.2.12  Table  2-32 

2.2.14  Summary 

2.2.15  Summary 

R39.  Guidance  for  retention  of 
records 

2.3.3 

Table  H-1.  Traceability  Matrix  to  SEI  CAF,  continued 


continued  on  next  page 


A-82 


SECMM-94-06ICMU/SEI-96-HB-004  v1 .1 


Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  Continued 


Table  H-1  Notes  *Items  preceded  by  are  suggestions  to  satisfy  this  requirement. 

There  is  an  implicit  assumption  which  should  he  explicit  (experience, 
"how  to"). 

**Item  where  SAM  deviates  acceptably  from  CAF  by  re-allocating 
requirement 
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Appendix  I:  References 


Introduction 


Reference  List 


This  appendix  provides  the  references  for  documents  cited  within  the 
SAM. 
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Capability  Maturity  Model,  Version  1.0,  (SECMM- 
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Carnegie  Mellon  University,  Software  Engineering 
Institute,  December  1994. 

[CAP]  Masters,  S.  CMM  Appraisal  Framework,  Version 
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Pittsburgh,  PA:  Carnegie  Mellon  University, 
Software  Engineering  Institute,  ADA2933()06,  1995. 
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Appendix  J:  SAM  Questionnaire  Context 


Introduction 


In  this  appendix 


This  appendix  contains  the  instructions  and  forms  for  the  SAM 
questionnaire. 


The  following  table  provides  a  guide  to  the  information  found  in  this 
appendix. 


Topic 

See  Page 

Instructions  for  the  SAM  Questionnaire 

A-88 

Glossary 

A-90 

Respondent  Feedback 

A-97 

Site  Coordinator  Instructions  for  Distributing 
Questionnaire 

A-98 

Questionnaires  by  Process  Area 

A-87 

continued  on  next  page 
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Instructions  for  the  SAM  Questionnaire 


Purpose 


Respondent 

information 


Questionnaire 

structure 


The  purpose  of  this  questionnaire  is  to  gather  preliminary  data  on  your 
organization’s  systems  engineering  process  capability  for  the  upcoming 
systems  engineering  appraisal. 


Please  identify  yourself  and  your  project  or  team,  as  appropriate. 

Name: _  Date:  _ 

Project: _ _ 


Following  this  introduction  is  a  glossary  of  terms  used  in  the 
questionnaire. 

Following  the  glossary  is  a  place  for  you  to  provide  feedback  on  this 
questionnaire  to  the  developers  of  the  SE-CMM  appraisal  method. 
Please  complete  this  form  ^ter  you  have  completed  the  questionnaire. 
Your  comments  drive  improvements  to  the  appraisal  method  and 
questionnaire. 

The  body  of  the  questionnaire  has  three  pages  of  questions  for  each  of 
the  17  process  areas  of  the  Systems  Engineering  Capability  Maturity 
Model  (SE-CMM).  Each  process  area  begins  with  a  summary 
description  and  a  list  of  its  base  practices.  The  base  practices  are 
followed  by  three  series  of  questions  (parts  2,  3,  and  4),  each  series 
addressing  a  different  perspective  (performing  the  work,  managing  the 
process,  and  infrastmcture  support). 


continued  on  next  page 
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Instructions  for  the  SAM  Questionnaire,  Continued 


Instructions 


Before  You  Begin 

Please  read  the  glossary  to  become  familiar  with  how  specific  terms  are 
used  in  the  questionnaire.  Different  organizations  have  different  internal 
meanings  for  common  and  uncommon  terms.  The  glossary  provides  a 
context  for  you  to  understand  the  intended  meaning  of  the  terms 
throughout  your  appraisal.  Then  follow  these  steps  for  each  process 
area. 


Step 

Action 

1 

Read  the  process  area  summary. 

'2 

Identify  the  base  practices  that  are  performed  on  your  project 
with  a  V  in  the  “Yes”  box.  Also  indicate  “No”  or  not 
applicable  (N/A).  (part  1) 

3 

If  you  had  no  “Yes”  answers,  please  proceed  directly  to  the 
next  process  area. 

4 

Answer  the  questions  that  follow  (parts  2,  3,  and  4)  from  the 
perspective  of  the  practices  for  which  you  answered  “Yes”  in 
part  1. 

Table  J-1.  Steps  for  Process  Areas. 


When  You  Are  Finished 

Please  fill  out  the  Respondent  Feedback  form.  Feel  free  to  comment  on 
both  content  and  format,  being  as  specific  as  possible.  Thank  you  for 
taking  the  time  to  fill  out  this  questionnaire. 
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Glossary 


Introduction 


Organizational 

terms 


This  glossary  defines  terms  used  in  the  questionnaire.  It  is 
recommended  that  you  read  through  theses  definitions  before  you  begin 
answering  the  questions  in  the  questionnaire. 


This  table  defines  how  the  terms  “organization”  and  “project”  are  used 
in  the  questionnaire. 


Term 

Definition 

exploratory 

questions 

Questions  that  are  open-ended  (e.g.,  begin  with  "would  you 
plan  ?  ...").  They  are  asked  of  interviewees  to  explain  or 
clarify  questionnaire  answers. 

interview 

script 

The  Appendix  E,  p.A-47  approach  bullets  which  includes 
exploratory  questions. 

organization 

In  the  context  of  the  SE-CMM,  “organization”  refers  to  the 
business  entity  being  appraised.  That  entity  should  have 
been  defined  for  you  by  those  who  gave  you  this 
questionnaire. 

Specifically,  an  organization  is  a  unit  within  a  company  or 
other  entity,  e.g.,  government  agency  or  branch  of  service, 
within  which  many  projects  are  managed  as  a  whole.  (All 
projects  within  an  organization  share  common  policies  at  the 
top  of  the  reporting  structure.) 
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project 


In  the  context  of  the  SE-CMM,  a  project  is  an  entity  within 
an  organization  that  produces  system  engineering  work 
products,  which  are  typically  associated  with  a  particular 
deliverable  system. 

The  project  is  the  aggregate  of  effort  and  other  resources 
focused  on  developing  and/or  maintaining  a  specific 
product.  The  product  may  include  hardware,  software,  and 
other  components.  T)^ically  a  project  has  its  own  funding, 
cost  accounting,  and  delivery  schedule.  Some  projects  are 
organized  around  teaming  structures.  Consult  your  site 
coordinator  for  interpretation  of  project  if  your  experience 
appears  inconsistent  with  the  above  definition. 

Some  aspects  of  systems  engineering  process  are  associated 
with  the  project  and  some  with  the  organization.  Even 
though  you  may  primarily  work  in  one  of  these  contexts  or 
both,  answer  the  questions  based  on  your  best  knowledge  of 
what  is  happening  in  the  context  in  question. 


Table  J-2.  Organizational  Terms. 


continued  on  next  page 
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Glossary,  continued 


SE-CMM  Appraisal  Method  acronym  expansions  and  glossary  definitions. 

Acronym  Expansion 

DoD  Department  of  Defense  which  sponsors  SEI’s 

involvement  in  EPIC 

DTIC  Defense  Technical  Information  Center  at  Cameron 

Station,  VA  in  Alexandria,  VA  from  which  this  and 
related  documents  can  be  ordered  by  US  Government 
and  contractors  with  accounts 

ESC  Electronic  Systems  Command  of  the  US  Air  Force 

(SETs  contracting  office  for  EPIC  support  funds 

ENS  (I  don’t  know  this  one  -  probably  a  typo) 

F&R  (I  don’t  know  this  one  either  -  let’s  check  the  context) 

FDRA  (Good  question  -  give  me  context) 

GP  Generic  Practice  -  a  practice  which  is  added  to  the  base 

practices  (BPs)  of  a  process  area  (PA)  to  create  a 
higher  process  capability  process.  See  the  SE-CMM 
chapter  4. 

GTE  General  Telephone  Company  which  provided  an  author 

for  SE-CMM 

Hanscom  AFB  Hanscom  Air  Force  Base  near  Boston,  MA.  Site  of 
ESC 

http  Action  command  which  initiates  a  World  Wide  Web 

connection 

NTIS  National  Technical  Information  System  run  by  the  US 

Department  of  Commerce  in  Springfield,  VA  from 
which  this  and  related  documents  can  be  purchased 

OSD  Office  of  the  Secretary  of  Defense  (US) 

PAOl  Process  Area  1 :  Analyze  Candidate  Solutions,  from 

SECMM 

PA _  Process  Area  (followed  by  number  between  01  and  18) 

from  SE-CMM 

RAl  (A  typo,  should  be  PAOl,  check  context  -  ref  to  trade 

studies  or  Analyzing  Candidate  Solutions  -  if  so, 
change  it) 

URL  (Unix  designator  for  files  used  on  World  Wide  Web 

sites  (check  context  could  also  be  typo  -  not  clear  why 
this  is  in  text!) 
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Glossary,  continued 


Expression 


Definition 


Affmitized 


Infeasible 

Iteratively 

Penalized 

Resequencing 

Substeps 

Wordsmith 


Verb  form  of  affinity  diagram  technique.  This  implies 
that  the  results  of  a  data  collection  exercise  have  been 
recorded  on  stickies  or  other  visual  medium  and  have 
been  clustered  according  to  underlying  factors  or 
themes.  They  may  or  may  not  have  been  labelled  yet, 
but  would  be  before  the  technique  is  completed. 

Not  do-able  with  the  resources  or  schedule  or  to  the 
minimum  quality  to  make  the  task  worth  doing 
Repeatedly,  as  in  “requirements  and  design  are 
performed  iteratively  until  they  converge  to  a  customer- 
satisfying  product  description  that  can  be  made  (has  a 
feasible  design,  development  schedule,  and  resources)” 
(misspelling  of  penalized  almost  surely  -  check  context) 
To  resort  in  temporal  (time)  order  as  in  an  agenda  or 
program  for  which  some  pre-requisites  have  not 
occurred,  so  other  later  activities  are  begun  instead 
Activities  performed  within  a  step  of  the  SECMM 
Appraisal  Method 

Cr^  exact  statements  with  exactly  the  intended 
meaning  rather  than  leave  an  expression  which  is 
“good  enough”  but  not  very  exact 
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Glossary,  continued 


Process  terms  The  process  terms  in  the  table  below  are  arranged  in  the  order  that  they 
appear  in  the  questionnaire. 


Term 

Definition 

Common 
causes  of 
variation 

Common  causes  are  causes  of  natural  variation  inherent 
in  a  process.  Removing  common  causes  of  variation 
involves  making  changes  to  the  process  itself. 

Defect  review 

A  review  of  a  work  product,  interim  or  deliverable,  that 
occurs  prior  to  the  release  of  the  work  product  to  the 
next  process  step.  The  review  involves  the  creator  of 
the  product  and  subject  matter  peers  who  identify 
defects  in  the  product  that  would  make  it  unsuitable  for 
use  in  the  next  work  process.  It  is  a  form  of  static 
testing  of  the  work  product. 

Organization's 

standard 

process 

(family  of 
processes) 

A  process  described  at  the  organizational  level  for  use 
by  projects  in  the  organization.  It  may  be  a  family  of 
processes  in  order  to  capture  the  different  classes  of 
processes  that  frequently  occur  in  organizations.  It  is 
intended  that  the  standard  process  be  tailored  into  a 
defined  process  to  meet  the  needs  of  specific  projects. 

Process 

Any  specific  combination  of  machines,  tools,  methods, 
materials,  and/or  people  employed  to  attain  specific 
qualities  in  a  product  or  service. 

Process  area 

A  set  of  practices  (i.e.,  process  requirements)  that 
address  the  same  purpose. 

Process 

capability 

At  its  simplest,  process  capability  indicates  the  range  of 
results  expected  by  performing  a  process.  Process 
capability  implies  competence.  Initially  a  process  is 
chaotic.  (In  this  context,  chaotic  implies  a  high  degree 
of  variation.)  It  is  stable  when  special  causes  of 
variation  have  been  removed,  and  capable  when 
common  causes  of  variation  have  been  reduced  to  meet 
customer  validated  requirements  (specification  limits). 
This  is  the  process  improvement  framework  upon 
which  the  SE-CMM  process  dimension  is  based. 

Process 

performance 

data 

Measurements  that  are  used  to  manage  the  process  used 
on  your  project  or  in  your  organization.  These  are 
measures  of  the  actual  results  achieved  using  the 
process. 

Table  J-3.  Process  Terms. 


continued  on  next  page 
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Glossary,  Continued 


Process  terms, 
continued 


Term 

Definition 

Process  and 

product 

measures 

Note:  both  work  products  and  the  process  can  be 
measured. 

Product  measures  are  measurable  attributes  of  a 
product,  such  as  size  or  number  of  defects,  and 
generally  do  not  vary  over  time  (i.e.,  the  product 
measure  can  be  measured  at  any  time  and  get  the  same 
result). 

Process  measures  are  measurable  attributes  of  the 
process  used  to  produce  the  product,  such  as  resources 
expended  per  product  or  percent  rework.  (They  must 
be  measured  during  the  process;  after  the  process  has 
completed,  only  product  measures  may  be  taken.) 

Special  causes 
of  variation 

Special  causes  of  variation  are  assignable  to  people, 
places,  materials,  events,  etc.  They  are  causes  of 
variation  that  are  not  attributable  to  the  process  itself, 
although  they  may  be  attributable  to  some  aspect  of  its 
execution. 

This  process 

The  process  or  processes  that  your  project  or 
organization  uses  to  implement  the  process  area  about 
which  you  are  answering  questions. 

Work  products 

Anything  produced  by  a  process.  This  includes 
specifications,  documents,  engineering  drawings,  etc., 
not  just  the  product  delivered  to  the  ultimate  customer. 
Delivered  products  are  those  work  products  that  the 
customer  receives.  These  may  also  include 
specifications,  interim  documents,  prototypes,  etc.,  in 
addition  to  the  final  end  product  (the  deliverable 
system). 

Table  J-3.  Process  Terms,  continued 
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Glossary,  continued 


Process  terms, 
continued 


Term 

Definition 

Defined 

process 

The  operational  definition  of  a  set  of  activities.  A 
defined  process  is  well  characterized  and  undertood, 
and  is  described  in  terms  of  standards,  tools,  and 
methods. 

Note:  A  defined  process  is  developed  by  tailoring  the 
organization’s  standards  process  to  fit  the  specific 
characteristics  of  its  intended  use.  (See  also  standard 
process) 

Well-defined 

process 

A  process  with  inputs,  entry  criteria,  tasks, 
verifications,  outputs,  and  exit  criteria  that  are 
documented,  consistent,  and  complete.  [SPICE  - 
modified] 

Project’s 

defined 

process 

The  operational  definition  of  the  process  as  used  by  a 
specific  project.  Well  characterized  and  understood,  it 
is  described  in  terms  of  standards,  procedures,  tools, 
and  methods.  It  is  developed  by  tailoring  the 
organization’s  standard  process  to  fit  the  specific 
characteristics  of  the  project.  [SECMM] 

Process 

performance 

A  measure  of  actual  results  achieved  by  following  a 
process.  [SECMM] 

Performance 

The  degree  to  which  a  system  or  component 
accomplishes  its  designated  functions  within  given 
constraints,  such  as  speed,  accuracy,  or  memory 
usage.  [IEEE  90] 

Table  J-3.  Process  Terms 
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Respondent  Feedback 


Respondent 

information 


Feedback 


Please  identify  yourself  and  your  project. 

Name: _  Date:  _ 

Organization: _ 

Amount  of  time  spent  filling  out  questionnaire: _ 

PAs  you  were  given  to  answer  questions  about:  _ all 

If  not  all,  then  please  list  the  numbers  you  were  given: _ 


We  would  greatly  appreciate  your  comments  on  the  questionnaire.  The 
developers  of  the  SE-CMM  appraisal  method  will  use  this  feedback  to 
improve  the  SE-CMM,  the  appraisal  method,  and  the  questionnaire. 
Feel  free  to  comment  on  both  content  and  format;  the  more  specific,  the 
better.  We  are  specifically  interested  in  the  following  areas:  clarity  of 
instructions,  usefulness  of  the  glossary,  other  terms  that  need  to  be 
defined,  structure  of  the  questionnaire,  and  understandability  of  the 
questions.  Thank  you  for  your  time  and  input! 
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Site  Coordinator  Instructions  for  Distributing  Questionnaire 


Introduction 


Time  constraints 


Recommended 

administration 

approach 


The  SE-CMM  questionnaire  is  the  first  step  in  a  data  gathering  process 
designed  to  provide  the  managers  and  practitioners  with  insight  into  the 
organization’s  systems  engineering  practices.  When  determining  who 
should  fill  out  the  questionnaire,  it  is  important  to  choose  individuals 
who  will  provide  answers  that  represent  the  entire  project/organization. 


It  typically  takes  around  two  hours  for  an  experienced  project  lead-level 
systems  engineer  to  complete  the  entire  SE-CMM  questionnaire.  For  a 
subject  matter  expert  to  fill  out  a  single  process  area  set  of  items 
typically  takes  between  5-10  minutes.  Questionnaire  response  time  is 
reduced  when  the  respondents  have  immediate  access  to  a  site 
coordinator  who  understands  the  model/appraisal  method,  and  when  the 
entire  questionnaire  is  answered  in  one  sitting. 


To  m^imize  the  use  of  both  the  site  coordinator's  and  respondents' 
time,  it  is  recommended  that  one  or  two  "appointments"  be  set  up  in  a 
roorn  large  enough  to  accommodate  all  respondents  with  a  suitable 
writing  surface,  e.g.,  a  large  table  or  several  small  tables.  Respondents 
are  invited  to  schedule  themselves  for  one  of  the  two  appointments. 
(One  is  optimal,  because  then  all  respondents  hear  answers  to  questions 
together.  However,  at  many  sites  getting  the  individuals  needed  to 
answer  the  questions  at  the  same  meeting  is  often  difficult.)  Before  the 
individuals  start  filling  out  the  questionnaires,  the  site  coordinator 
introduces  the  respondents  to  the  model  and  appraisal,  distributes  the 
questionnaire,  and  makes  clear  that  the  answers  are  to  reflect  their 
individual  opinions.  He/she  also  makes  clear  that  the  site  coordinator  is 
there  to  clarify  terminology  and  concepts,  and  respondents  are 
encouraged  to  voice  questions.  The  site  coordinator  can  then  answer  to 
the  entire  group  assembled.  The  site  coordinator  also  records  any 
questions  he/she  cannot  answer  so  he/she  can  contact  an  SE-CMM 
facilitator  for  guidance. 

After  each  respondent  is  finished,  the  site  coordinator  collects  the 
questionnaire  and,  if  not  already  filled  in  by  respondent,  completes  the 
'time  spent’  portion  of  the  feedback  form. 


continued  on  next  page 
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Site  Coordinator  Distribution  Instructions,  Continued 


Questionnaire  To  maximize  the  accuracy  of  initial  responses  to  the  questions  in  the  SE- 
distribution  table  CMM  questionnaire,  it  is  recommended  that  the  questionnaires  be 
distributed  to  individuals  with  the  skills  and  roles  expressed  in  the 
following  table. 

Process  Area  Primary  Secondary 

Respondents  Respondents 

01:  Analyze  Systems  engineering  Any  senior  practitioner 

Candidate  Solutions  leads  for  the  projects  with  significant 

selected  for  appraisal  system  design 

experience 

02:  Derive  and  Systems  engineering 

Allocate  Requirements  leads  for  the  projects 

selected  for  appraisal 

03:  Evolve  System  Systems  engineering 

Architecture  leads  for  the  projects 

selected  for  appraisal 

04:  Integrate  Systems  engineering  Senior  specialty 

Disciplines  leads  for  the  projects  engineers  (e.g., 

selected  for  appraisal  reliability,  safety, 

manufacturing,  human 
factors)  working  on 
the  projects  selected 

_  for  appraisal 

05:  Integrate  System  Systems  engineering 

leads  for  the  projects 
selected  for  appraisal 

06:  Understand  Systems  engineering  •  Technical  marketing 

Customer  Needs  and  leads  for  the  projects  personnel 
Expectations  selected  for  appraisal  .  Proposal  personnel 

•  Customer  service 

_  personnel 

07:  Verify  and  Systems  engineering  System  verification 

Validate  System  leads  for  the  projects  manager  or  senior  test 

selected  for  appraisal  engineers 

Table  J-4.  Questionnaire  Distribution  Table. 

continued  on  next  page 
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Site  Coordinator  Distribution  Instructions,  continued 


Questionnaire 
distribution  table, 
continued 


Process  Area 

Primary 

Respondents 

Secondary 

Respondents 

08:  Ensure  Quality 

Senior  project-level 
quality  manager  or 
lead  (in  environments 
with  shared  quality 
leadership 

responsibility,  systems 
engineering  lead  for 
the  project) 

•  Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

•  Organizational 
quality  manager, 
total  quality 
management 
coordinator 

09:  Manage 
Configurations 

Senior  project-level 

CM  manager  for  the 
projects  selected  for 
appraisal 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

10:  Monitor  and 
Control  Technical 
Effort 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

11:  Plan  Technical 
Effort 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

12:  Manage  Risk 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Project  or  program 
manager  for  the 
projects  selected  for 
appraisal 

13:  Define 
Organization's 

Systems  Engineering 
Process 

Individuals 
responsible  for 
defining  organization 
level  processes;  may 
be  part  of  the  quality 
leadership  area, 
policies/procedures 
area,  or  other  support 
group 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Table  J>4.  Questionnaire  Distribution  Table,  continued 
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Site  Coordinator  Distribution  instructions,  continued 


Questionnaire 
distribution  table, 
continued 


Process  Area 

Primary 

Respondents 

Secondary 

Respondents 

14:  Improve 
Organization’s 

Systems  Engineering 
Processes 

Individuals 
responsible  for 
deploying  organization 
level  process 
improvement  activities; 
may  be  part  of  the 
quality  leadership  area, 
policies/procedures 
area,  or  other  support 
group 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

15:  Manage  Product 
Line  Evolution 

Individuals  at 
organization  level 
responsible  for 
strategic  product  line 
positioning  and 
advancement;  may  be 
in  R«&D,  technical 
marketing,  or  other 
support  stmcture 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

16:  Manage  Systems 
Engineering  Support 
Environment 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Individuals  at 
organization  level 
involved  in  deploying 
new  development 
technologies 

17:  Provide  Ongoing 
Skills  and  Knowledge 

Individuals 
responsible  for 
recruiting, 
subcontracting  or 
deployment  of 
organization-level 
training;  may  be  part 
of  an  R&D  group, 
training  department,  or 
other  support  stmcture 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

18:  Coordinate  with 
Suppliers 

Buyers,  subcontract 
managers  or  project 
leaders  of  projects 
with  teammates 

Systems  engineering 
leads  for  projects  with 
non-organizational 
subcontractors  or 
teammates. 

Table  J-4.  Questionnaire  Distribution  Table,  continued 
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Site  Coordinator  Distribution  Instructions,  continued 


Distribution 

recommendation 


Even  though  there  are  other  roles  called  out  as  primary  targets  for  the 
questionnaire  in  certain  instances,  having  the  SE  leads  answer  all  the 
PAs  provides  an  overall  context  of  how  things  appear  from  the  project 
viewpoint,  which  can  be  very  valuable.  Therefore,  it  is  recommended 
that  the  SE  leads  complete  all  the  PAs  whenever  feasible. 
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PA  01 :  Analyze  Candidate  Solutions 


The  purpose  of  Analyze  Candidate  Solutions  is  to  perform  studies  and  analyses  that 
will  result  in  the  selection  of  a  solution  to  meet  the  identified  problem  and  its 
defined  constraints.  Analyze  Candidate  Solutions  involves  defining  the  approach 
and  evaluation  criteria  for  the  analysis,  as  well  as  for  choosing,  selecting,  and 
studying  the  candidate  solutions.  It  also  involves  communicating  the  rationale  and 
results  of  the  analysis. 

1 .  B  a  s  e  Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 

practices  you  do  not  have  to  personally  be  involved  in  performing  the  practice  — 

it's  enough  that  you  know  who  performs  it. 

Comments: 

Yes  No  N/A 

[  ]  [  ]  [  ]  Establish  evaluation  criteria  based  on  the  identified  problem  and  its  defined 
constraints. 

[  ]  [  ]  [  ]  Define  the  general  approach  for  the  analysis,  based  on  the  established 
evaluation  criteria. 

[  ]  [  ]  [  ]  Identify  alternatives  for  evaluation  in  addition  to  those  provided  with  the 
problem  statement. 

[  ]  [  ]  [  1  Analyze  the  competing  candidate  solutions  against  the  established 
evaluation  criteria. 

[]  []  []  Select  the  solution  that  satisfies  the  established  evaluation  criteria. 

[  ]  [  ]  [  ]  Capture  the  disposition  of  each  alternative  under  consideration  and  the 
rationale  for  the  disposition. 


continued  on  next  page 


Process  area 
summary 


PA  01 :  Analyze  Candidate  Solutions,  continued 


2.  Performing  Indicate  (y,  no,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 
the  work  perfomied  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes"  marks 

Comments  ■  should  indicate  that  it  is  both  generally  and  visibly  practiced. 


ID 

Questions  relating  to  the  performance  of  the  base  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing  this 
process  area? 

2.2.2 

Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 

3,  Supporting  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  organizational  infrastructure  supports 
infrastructure  Analyzing  Candidate  Solutions  by  making  the  activities  related  to  these  processes 
^  .  visible  to  the  organization  as  a  whole. 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 

supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of 
processes)  for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  01 :  Analyze  Candidate  Solutions,  continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Analyze 

management  Candidate  Solutions  process  area  is  managed  using  the  following  practices. 

Comments: 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

y/n 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

■ 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing  this 
process  area's  work  products  and/or  services  for  your  project? 

■ 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in  place 
for  performing  this  process  area? 

■ 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

■ 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

1 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its  plan? 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 

■ 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

1 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

1 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

4.2.2 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process  capability? 

■ 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

■ 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

■ 

end  of  PA  01 :  Deriving  and  Allocating  Requirements 


PA  02:  Derive  and  Allocate  Requirements 


Process  area  The  purpose  of  Derive  and  Allocate  Requirements  is  to  analyze  the  system  and  other 

summary  requirements  and  derive  a  more  detailed  and  precise  set  of  requirements.  These 

derived  requirements  are  allocated  to  system  functions;  objects;  people;  and 
supporting  processes,  products,  and  services,  which  can  be  used  to  synthesize 
solutions.  This  process  area  addresses  both  the  analysis  of  system-level 
requirements  and  the  allocation  of  system-level  or  derived  requirements  to  lower 
level  functions  or  objects.  This  involves  addressing  the  concept  of  operations, 
functional  partitioning,  object  identification,  and  performance  allocation,  as  well  as 
capturing  the  status  and  traceability  of  requirements.  The  derived  and  allocated 
requirements  will  evolve  as  the  systems  requirements  evolve  over  time.  When 
corrective  actions  have  an  impact  on  requirements,  it  may  be  necessary  to  revise  the 
derived  and  allocated  requirements. 

1 .  Base  Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 

practices  you  do  not  have  to  personally  be  involved  in  performing  the  practice  — 

Qpmments:  it's  enough  that  you  know  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Develop  a  detailed  operational  concept  of  the  interaction  of  the  system,  the 
user,  and  the  environment,  that  satisfies  the  operational  need. 

[  1  [  ]  [  ]  Identify  key  requirements  that  have  a  strong  influence  on  cost,  schedule, 
functionality,  risk,  or  performance. 

[  ]  [  ]  [  ]  P^ition  requirements  into  groups  of  requirements  based  on  established 
criteria,  such  as  similar  functionality,  performance,  or  coupling,  to 
facilitate  and  focus  the  requirements  analysis. 

[  ]  [  ]  [  ]  Derive,  from  the  system  and  other  (e.g.,  environmental)  requirements, 
requirements  that  may  be  logically  inferred  and  implied  as  essential  to 
system  effectiveness. 

[  ]  [  ]  [  ]  Identify  the  requirements  associated  with  external  interfaces  to  the  system 
and  interfaces  between  functional  partitions  or  objects. 

[  ]  [  ]  [  ]  Allocate  requirements  to  function^  partitions,  objects,  people,  or  support 
elements  to  support  synthesis  of  solutions. 

[]  []  []  Analyze  requirements  to  ensure  that  they  are  verifiable  by  the  methods 
available  to  the  development  effort. 

[  ]  [  ]  [  ]  Maintain  requirements  traceability  to  ensure  that  lower  level  (derived) 

requirements  are  necessary  and  sufficient  to  meet  the  objectives  of  higher 
level  requirements. 

[  ]  [  ]  [  ]  Capture  system  and  other  requirements,  derived  requirements,  derivation 
rationale,  allocations,  traceability,  and  requirements  status. 

Feel  free  to  add  clarifying  comments  in  the  margin. 
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PA  02:  Derive  and  Allocate  Requirements,  continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes"  marks 
„  should  indicate  that  it  is  both  generally  and  visibly  practiced. 

CoTTifnents: 


ID 

Questions  relating  to  the  performance  of  the  base  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing  this 
process  area? 

2.2.2 

Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 

3.  Supporting  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  organizational  infrastructure 
infrastructure  supports  Derive  and  Allocate  Requirements  by  making  the  activities  related  to 
^  these  processes  visible  to  the  organization  as  a  whole. 

Comments:  ^ 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 
supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of 
processes)  for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  02:  Derive  and  Allocate  Requirements,  continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Derive 
management  and  Allocate  Requirements  process  area  is  managed  using  the  following 
^  practices. 

Comments:  ^ 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

ly/nl 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

■ 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing  this 
process  area's  work  products  and/or  services  for  your  project? 

■ 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in  place 
for  performing  this  process  area? 

■ 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

■ 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

1 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

Does  your  project  measure  the  progress  of  this  process  area  against  its  plan? 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 

■ 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

1 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

1 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

m 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process  capability? 

■ 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

■ 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

■ 

end  of  PA  02:  Derive  and  Allocate  Requirements 


PA  03:  Evolve  System  Architecture 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Evolve  System  Architecture  is  to  provide  a  basis  for  establishing 
and  evolving  a  system  design.  It  involves  deriving  the  architecture  requirements, 
identifying  key  design  issues,  determining  the  functional  and  physical  stmcture  and 
interfaces,  and  allocating  the  architecture  requirements  to  system  elements.  The 
practices  described  herein  are  expected  to  be  performed  iteratively  with  other 
systems  engineering  practices  until  the  architecture  is  handed  off  to  the 
implementing  or  component  engineering  disciplines. 

System  architecture  comprises  functional  (or  logical),  physical  (tangible),  and 
foundation  architectures.  Evolve  System  Architecture  activities  are  applicable  to  all 
life-cycle  phases  of  a  product  and  may  be  initiated  either  by  new  development, 
changes  in  requirements,  or  corrective  actions. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 

you  do  not  have  to  personally  be  involved  in  performing  the  practice  — 

it's  enough  that  you  know  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Derive  the  requirements  for  the  system  architecture. 

[  ]  [  ]  [  ]  Identify  the  key  design  issues  that  must  be  resolved  to  support  successful 
development  of  the  system. 

[  ]  [  ]  [  ]  Generate  altemative(s)  and  constraints  for  the  architecture,  and  select  a 

solution  in  accordance  with  the  Analyze  Candidate  Solutions  process  area 
(PAOl). 

[  ]  [  ]  [  ]  Develop  the  interface  requirements  for  the  chosen  architecture 
components. 

[  ]  [  ]  [  ]  Allocate  the  system  and  derived  requirements  to  the  chosen  architecture 
components  and  interfaces. 

[  ]  [  ]  [  ]  Maintain  requirement  traceability  for  the  architecture's  requirements  to 

ensure  that  lower  level  (derived)  requirements  are  necessary  and  sufficient 
to  meet  the  needs  of  higher  level  requirements  or  design. 

[  ]  [  ]  [  ]  Describe  the  system  architecture  by  capturing  the  design  results  and 
rationale. 

[  ]  [  ]  [  ]  Identify  appropriate  derived  requirements  that  address  the  effectiveness 
and  cost  of  life-cycle  phases  following  development,  such  as  production 
and  operation.. 

Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  03:  Evolve  System  Architecture,  Continued 


2.  Performing 
the  work 

Comments: 


Indicate  (y,  no,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes"  marks 
should  indicate  that  it  is  both  generally  and  visibly  practiced. 


ID  Questions  relating  to  the  performance  of  the  base  practices. 

2.2. 1  Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing  this 
process  area? 

2.2.2  Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1  Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2  Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1  Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2  Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2. 1  Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 


3.  Supporting 
infrastructure 

Comments: 


Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  organizational  infrastructure 
supports  Evolve  System  Architecture  by  making  the  activities  related  to  these 
processes  visible  to  the  organization  as  a  whole. 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 

supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of 
processes)  for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  03:  Evolve  System  Architecture,  continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Evolve 

management  System  Architecture  process  area  is  managed  using  the  following  practices. 

Comments: 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

y/n 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

■ 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 

■ 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

1 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

1 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

■ 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

1 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

■ 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

■ 

end  of  PA  03:  Evolve  System  Architecture 


PA  04:  Integrate  Disciplines 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Integrate  Disciplines  is  to  identify  those  disciplines  necessary  for 
effective  system  development  and  create  an  environment  in  which  they  jointly  and 
effectively  work  together  toward  a  common  agenda.  Each  discipline’s  unique 
expertise  and  concerns  are  brought  forward  and  considered,  but  the  focus  on  total 
system  development  is  maintained.  These  disciplines  may  include,  but  are  not 
limited  to,  problem  domain,  marketing,  manufacturing,  component  design, 
development,  reliability,  maintainability,  operations,  quality,  supportability,  human 
factors,  logistics,  safety,  and  security.  It  is  critical  to  be  able  to  meld  such 
disciplines  without  sacrificing  their  parochial  interests  concerning  issues  important 
to  and  unique  to  each  discipline.  This  cooperative  environment  must  persist 
throughout  the  system  life  cycle. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 

you  do  not  have  to  personally  be  involved  in  performing  the  practice  - 

it's  enough  that  you  know  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Involve  the  disciplines  that  are  essential  to  system  development  in  a  timely 
manner. 

[  ]  [  ]  [  ]  Promote  cross-discipline  understanding  among  the  developers. 

[  ]  [  ]  [  ]  Establish  methods  for  interdisciplinary  coordination. 

[  ]  [  ]  [  ]  Establish  and  use  methods  for  identifying  and  resolving  interdisciplinary 

issues,-and  creating  integrated  solutions. 

[  ]  [  ]  [  ]  Communicate  results  of  interdisciplinary  activities  to  affected  groups. 

[  ]  [  ]  [  ]  Develop  project  goals  and  ensure  that  all  affected  groups  and  individuals 

are  fully  aware  of  them. 

Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  04:  Integrate  Disciplines,  continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes"  marks 
should  indicate  that  it  is  both  generally  and  visibly  practiced. 

Comments: 


ID 

Questions  relating  to  the  performance  of  the  base  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing 
this  process  area? 

2.2.2 

Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 

3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastructure 
infrastructure  supports  Integrate  Disciplines  by  making  the  activities  related  to  these 
processes  visible  to  the  organization  as  a  whole. 

Comments: 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 

supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of 
processes)  for  this  process  area? 

■ 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  04:  Integrate  Disciplines,  continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Integrate 

management  Disciplines  process  area  is  managed  using  the  following  practices. 

Comments: 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

U/n 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

■ 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

■ 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

■ 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

■ 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

1 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

■ 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  Aat  planned? 

■ 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

1 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

1 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

1 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

■ 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

■ 

end  of  PA  04:  Integrate  Disciplines 


PA  05:  Integrate  System 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Integrate  System  is  to  ensure  that  system  elements  will  function  as  a 
whole.  This  primarily  involves  identifying,  defining,  and  controlling  interfaces,  as 
well  as  verifying  system  functions  that  require  multiple  system  elements.  The 
activities  associated  with  Integrate  System  occur  throughout  the  entire  product  life 
cycle. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
it's  enough  that  you  know  who  performs  it. 


Yes  No  N/A 

[  ]  [  ]  [  ]  Develop  detailed  specifications  of  the  interfaces  implied  by  the  system 


architecture. 

[  ]  [  ]  [  ]  Coordinate  interface  specifications  and  changes  with  all  affected  groups 
and  individuals. 

[  ]  [  ]  [  ]  Verify  the  receipt  of  each  system  element  required  to  assemble  the  system 
in  accordance  with  the  physical  architecture. 

[  ]  [  ]  [  ]  Verify  the  implemented  design  features  of  developed  or  purchased  system 
elements  against  their  requirements. 

[  ]  [  ]  [  ]  Verify  that  the  system  element  interfaces  comply  with  the  interface 
specifications  prior  to  assembly. 

[  ]  [  ]  [  3  Assemble  aggregates  of  system  elements  in  accordance  with  the 
established  integration  strategy. 

[  ]  [  ]  [  ]  Check  the  integrated  system  interfaces  in  accordance  with  the  established 


integration  strategy.  . 

[  ]  [  ]  [  ]  Develop  an  integration  strategy  and  supporting  documentation  that  identify 
the  optimal  sequence  for  receipt,  assembly,  and  activation  of  the  various 
components  that  make  up  the  system. 


Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  05:  Integrate  System,  continued 


2.  Performing 
the  work 

Comments: 


Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes"  marks 
should  indicate  that  it  is  both  generally  and  visibly  practiced. 


ID 

Questions  relating  to  the  performance  of  the  base  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing  this 
process  area? 

2.2.2 

Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area’s  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area’s  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area’s  work  products? 

3.  Supporting  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  organizational  infrastructure  supports 
infrastructure  Integrate  System  by  making  the  activities  related  to  these  processes  visible  to  the 
^  organization  as  a  whole. 

Comments: 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 

supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of 
processes)  for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  05:  Integrate  System,  Continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Integrate 
management  System  process  area  is  managed  using  the  following  practices. 

Comments: 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

y/n 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

□ 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

■ 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  Aat  planned? 

■ 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

1 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

1 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

4.2.2 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

1 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

■ 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

■ 

end  of  PA  05:  Integrate  System 


PA  06:  Understand  Customer  Needs  and  Expectations 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Understand  Customer  Needs  and  Expectations  is  to  elicit,  stimulate, 
analyze,  and  communicate  customer  needs  and  expectations  to  obtain  a  better 
understanding  of  what  will  satisfy  the  customer.  Understand  Customer  Needs  and 
Expectations  involves  engaging  the  customer  or  surrogate  in  ongoing  dialogue 
designed  to  translate  his/her  needs  and  expectations  into  a  verifiable  set  of 
requirements  which  the  customer  understands  and  which  provide  the  basis  for 
agreements  between  the  customer  and  the  systems  engineering  effort. 

Customer  needs  typically  change  over  time.  Organizations  need  to  have  a  workable 
way  to  incorporate  such  changes  into  current  and  future  versions  of  the  product. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 

you  do  not  have  to  personally  be  involved  in  performing  the  practice  — 

it's  enough  that  you  know  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Elicit  the  customer  needs,  expectations,  and  measures  of  effectiveness. 

[]  []  []  Analyze  the  customer  needs  and  expectations  to  develop  a  preliminary 
operational  concept  of  the  system. 

[  ]  [  ]  [  ]  Develop  a  statement  of  system  requirements. 

[  ]  [  ]  [  ]  Obtain  the  customers'  agreement  that  system  requirements  satisfy  their 
needs  and  expectations 

[  ]  [  ]  [  ]  Inform  the  customer  on  a  regular  basis  about  the  status  and  disposition  of 
needs,  expectations,  and  measures  of  effectiveness. 

Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  06:  Understand  Customer  Needs  and  Expectations,  continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes"  marks 

should  indicate  that  it  is  both  generally  and  visibly  practiced. 

Comments: 


ID 

Questions  relating  to  the  performance  of  the  base  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing 
this  process  area? 

2.2.2 

Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 

3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastructure 

infrastructure  supports  Understand  Customer  Needs  and  Expectations  by  making  the  activities 
related  to  these  processes  visible  to  the  organization  as  a  whole. 

Comments: 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 
supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of  processes) 
for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  06:  Understand  Customer  Needs  and  Expectations,  Continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the 

management  Understand  Customer  Needs  and  Expectations  process  area  is  managed  using  the 
^  following  practices. 

Comments:  ^  ^ 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

■ 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

■ 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

■ 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

■ 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

1 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

■ 

2.4.2 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 

■ 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

1 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

1 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

■ 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

1 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

■ 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

■ 

end  of  PA  06:  Understand  Customer  Needs  and  Expectations 


PA  07:  Verify  and  Validate  System 


The  purpose  of  Verify  and  Validate  System  is  to  ensure  that  the  developer/supplier 
team  performs  increasingly  comprehensive  evaluations  to  ensure  that  evolving  work 
products  will  meet  all  requirements.  The  activities  associated  with  Verify  and 
Validate  System  begin  early  in  the  development,  address  all  work  products 
(including  requirements  and  design),  and  continue  through  development  and 
integration  of  system  elements  into  production,  use,  and  disposal  of  the  system. 

The  scope  of  verification  covers  development  of  the  full  system,  as  well  as  its 
production,  operation,  and  support.  Validation  is  a  measure  of  customer 
satisfaction,  given  the  customer's  operational  need.  Validation  should  continue 
throughout  product  use. 

1 .  Base  Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 

practices  you  do  not  have  to  personally  be  involved  in  performing  the  practice 

Comments:  —  it's  enough  that  you  know  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Establish  plans  for  verification  and  validation  that  identify  the  overall 
requirements,  objectives,  resources,  facilities,  special  equipment,  £ind 
schedule  applicable  to  the  system  development. 

[  ]  [  ]  [  ]  Define  the  methods,  process,  reviews,  inspections  and  tests  by  which 
incremental  products  are  verified  against  established  criteria  or 
requirements  that  were  established  in  a  previous  phase. 

[  ]  [  ]  [  ]  Define  the  methods,  process,  and  evaluation  criteria  by  which  the 
system  or  product  is  verified  against  the  system  or  product 
requirements. 

[  ]  [  ]  [  ]  Define  the  methods,  process,  and  evaluation  criteria  by  which  the 

system  or  product  will  be  validated  against  the  customer’s  needs  and 
expectations. 

[  ]  [  ]  [  ]  Perform  the  verification  and  validation  activities  that  are  specified  by  the 
verification  and  validation  plans  and  procedures,  and  capture  the  results. 

[  ]  [  ]  [  ]  Compare  the  collected  test,  inspection,  or  review  results  with 
established  evaluation  criteria  to  assess  the  degree  of  success. 

Feel  free  to  add  clarifying  comments  in  the  margin. 


Process  area 
summary 


continued  on  next  page 


PA  07:  Verify  and  Validate  System,  continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 
the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes"  marks 
Comment  ■  should  indicate  that  it  is  both  generally  and  visibly  practiced. 


ID  Questions  relating  to  the  performance  of  the  base  practices. 

2.2. 1  Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing 
this  process  area? 

2.2.2  Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1  Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2  Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2. 1  Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2  Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1  Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 


3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastructure  supports 
infrastructure  Verify  and  Validate  System  by  making  the  activities  related  to  these  processes 
-  ,  visible  to  the  organization  as  a  whole. 

Comments:  ^ 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 
supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of  processes) 
for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  07:  Verify  and  Validate  System,  continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Verify 

management  and  Validate  System  process  area  is  managed  using  the  following  practices. 

Comments: 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

y/n 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

■ 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

■ 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

■ 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

■ 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

1 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

4.2.1 

Does  your  project  or  orgmization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

end  of  PA  07:  Verify  and  Validate  System 


PA  08:  Ensure  Quality 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Ensure  Quality  is  to  address  not  only  the  quality  of  the  system,  but 
also  the  quality  of  the  process  being  used  to  create  the  system  and  the  degree  to 
which  the  project  follows  its  defined  process.  The  underlying  concept  of  this 
process  area  is  that  high-quality  systems  can  only  be  consistently  produced  on  a 
continuous  basis  if  a  process  exists  to  continuously  measure  and  improve  quality. 
In  addition,  this  process  must  be  adhered  to  rigorously  and  throughout  the  system 
life  cycle.  Key  aspects  of  the  process  required  to  develop  high-quality  systems  are 
measurement,  analysis,  and  corrective  action. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 

you  do  not  have  to  personally  be  involved  in  performing  the  practice 

"  it's  enough  that  you  know  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Ensure  the  defined  system  engineering  process  is  adhered  to  during  the 
system  life  cycle. 

[  ]  [  ]  [  ]  Evaluate  work  product  measures  against  the  requirements  for  work 
product  quality. 

[  ]  [  ]  [  ]  Measure  the  quality  of  the  systems  engineering  process  used  by  the 
project. 

[  ]  [  ]  [  ]  Analyze  quality  measurements  to  develop  recommendations  for  quality 
improvement  or  corrective  action  as  appropriate. 

[  ]  [  ]  [  ]  Obtain  employee  participation  in  identifying  and  reporting  quality 
issues. 

[  ]  [  ]  [  ]  Initiate  activities  that  address  identified  quality  issues  or  quality 
improvement  opportunities. 

[  ]  [  ]  [  ]  Establish  a  mechanism  or  a  set  of  mechanisms  to  detect  the  need  for 
corrective  actions  to  processes  or  products. 

Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  08:  Ensure  Quality,  continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes"  marks 

^  should  indicate  that  it  is  both  generally  and  visibly  practiced. 

Comments: 


ID 

Questions  relating  to  the  performance  of  the  base  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing 
this  process  area? 

2.2.2 

Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area’s  work  products? 

3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastructure 
infrastructure  supports  Ensure  Quality  by  making  the  activities  related  to  these  processes 
^  visible  to  the  organization  as  a  whole. 

Comments: 


ID 

Questions  relating  to  the  organization’s  infrastructure 
which  supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family 
of  processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the 
work  products  resulting  from  following  its  standard  process  (or  family 
of  processes)  for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for 
this  process  area? 

■ 

continued  on  next  page 


PA  08:  Ensure  Quality,  continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Ensure 

management  Quality  process  area  is  managed  using  the  following  practices. 

Comments: 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

ly/n 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

■ 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

■ 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

■ 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

■ 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

1 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

■ 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 

■ 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

1 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

1 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

1 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

■ 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

■ 

end  of  PA  08:  Ensure  Quality 


PA  09:  Manage  Configurations 


Process  area  The  purpose  of  Manage  Configurations  is  to  maintain  data  on  and  status  of 
summary  identified  configuration  units,  and  to  analyze  and  control  changes  to  the  system  and 

its  configuration  units.  Managing  the  system  configuration  involves  providing 
accurate  and  current  configuration  data  and  status  to  developers  and  customers. 

This  process  area  is  applicable  to  all  work  products  that  are  placed  under 
configuration  management.  An  example  set  of  work  products  that  may  be  placed 
under  configuration  management  could  include  hardware  and  software 
configuration  items,  design  rationale,  requirements,  product  data  files,  or  trade 
studies. 


1 .  Base 
practices 

Comments: 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
—  it's  enough  that  you  know  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Decide  among  candidate  methods  for  configuration  management. 

[  ]  [  ]  [  ]  Identify  configuration  units  that  constitute  identified  baselines. 

[  ]  [  ]  [  ]  Maintain  a  repository  of  work  product  baselines. 

[  ]  [  ]  [  ]  Control  changes  to  established  configuration  units. 

[  ]  [  ]  [  ]  Communicate  status  of  configuration  data,  proposed  changes,  and 
access  information  to  affected  groups. 

Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  09:  Manage  Configurations,  continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 
the  work  performed  as  part  of  doing  die  work  associated  with  this  process  area.  "Yes"  marks 
Comments-  should  indicate  that  it  is  both  generally  and  visibly  practiced. 


ID 

Questions  relating  to  the  performance  of  the  base  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing  this 
process  area? 

2.2.2 

Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 

3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastructure  supports 
infrastructure  Manage  Configurations  by  making  the  activities  related  to  these  processes  visible  to 
_  the  organization  as  a  whole. 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 

supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of  processes) 
for  this  process  area? 

■ 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  09:  Manage  Configurations,  continued 


4.  Process  Indicate  (y ,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Manage 

management  Configurations  process  area  is  managed  using  the  following  practices. 

Comments: 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

y/n 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

■ 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

■ 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

■ 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

■ 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

1 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

in 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

ii 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

end  of  PA  09:  Manage  Configurations 


PA  10:  Manage  Risk 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Manage  Risk  is  to  identify,  assess,  monitor,  and  mitigate  risks  to 
the  success  of  both  the  systems  engineering  activities  and  the  overall  technical 
effort.  This  process  area  continues  throughout  the  life  of  the  project.  Similar  to  the 
Plan  Technical  Effort  (PA12)  and  Monitor  and  Control  Technical  Effort  (PAl  1) 
process  areas,  the  scope  of  this  process  area  includes  both  the  systems  engineering 
activities  and  the  overall  technical  project  effort,  as  the  systems  engineering  effort 
on  the  project  cannot  be  considered  successful  unless  the  overall  technical  effort  is 
successful. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 

you  do  not  have  to  personally  be  involved  in  performing  the  practice 

—  it's  enough  that  you  know  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Develop  a  plan  for  risk-management  activities  that  is  the  basis  for 

identifying,  assessing,  mitigating,  and  monitoring  risks  for  the  life  of 
the  project. 

[  ]  [  ]  [  ]  Identify  project  risks  by  examining  project  objectives  with  respect  to  the 
alternatives  and  constraints  and  identifying  what  can  go  wrong. 

[  ]  [  ]  [  ]  Assess  risks  and  determine  the  probability  of  occurrence  and 
consequence  of  realization. 

[  ]  [  ]  [  ]  Obtain  formal  recognition  of  the  project  risk  assessment. 

[]  []  []  Implement  the  risk  mitigation  activities. 

[  ]  [  ]  [  ]  Monitor  risk  mitigation  activities  to  ensure  the  desired  results  are  being 
obtained. 


Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  10:  Manage  Risk,  continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes"  marks 
„  should  indicate  that  it  is  both  generally  and  visibly  practiced. 

Comments: 


ID 

Questions  relating  to  the  performance  of  the  base  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing 
this  process  area? 

2.2.2 

Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 

3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastructure 
infrastructure  supports  Manage  Risk  by  m^ing  the  activities  related  to  these  processes 
^  visible  to  the  organization  as  a  whole. 

Comments: 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 

supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of 
processes)  for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  10:  Manage  Risk,  continued 


4.  Process 
management 


Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Manage 
Risk  process  area  is  managed  using  the  following  practices. 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

|y/n| 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

■ 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

■ 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

■ 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

■ 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

■ 

m 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 

■ 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

1 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

1 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

■ 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

1 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

■ 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

■ 

end  of  PA  10:  Manage  Risk 


PA  1 1 :  Monitor  and  Control  Technical  Effort 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Monitor  and  Control  Technical  Effort  is  to  provide  adequate 
visibility  of  actual  progress  and  risks.  Visibility  encourages  timely  corrective  action 
when  performance  deviates  significantly  from  plans. 

Monitor  and  Control  Technical  Effort  involves  directing,  tracking  and  reviewing  the 
project's  accomplishments,  results,  and  risks  against  its  documented  estimates, 
commitments,  and  plans.  A  documented  plan  is  used  as  the  basis  for  tracking  the 
activities  and  risks,  communicating  status,  and  revising  plans. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
-  it's  enough  that  you  know  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Direct  technical  effort  in  accordance  with  technical  management  plans. 

[  ]  [  ]  [  ]  Track  actual  use  of  resources  against  technical  management  plans. 

[  ]  [  ]  [  ]  Track  performance  against  the  established  technical  parameters. 

[  ]  [  ]  [  ]  Review  performance  against  the  technical  management  plans. 

[  ]  [  ]  [  ]  Analyze  issues  resulting  from  tracking  and  review  of  technical 
parameters  to  determine  corrective  actions. 

[  ]  [  ]  [  ]  Take  corrective  actions  when  actual  results  deviate  from  plans. 


Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  11:  Monitor  and  Control  Technical  Effort,  Continued 


2.  Performing 
the  work 

Comments: 


Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes" 
marks  should  indicate  that  it  is  both  generally  and  visibly  practiced. 


ID  Questions  relating  to  the  performance  of  the  base  practices. 

2.2.1  Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing 
this  process  area? 

2.2.2  Does  your  project  place  this  process  area’s  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1  Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2  Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1  Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2  Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1  Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 


3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastructure 

infrastructure  supports  Monitor  and  Control  Technical  Effort  by  making  the  activities  related 
_  to  these  processes  visible  to  the  organization  as  a  whole. 

Comments:  ^  ^ 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 

supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of 
processes)  for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  11:  Monitor  and  Control  Technical  Effort,  Continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Monitor 
management  and  Control  Technical  Effort  process  area  is  managed  using  the  following 
^  practices. 

Comments: 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

y/n 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

ii 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  ^at  planned? 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

end  of  PA  1 1 :  Monitor  and  Control  Technical  Effort 


PA  12:  Plan  Technical  Effort 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Plan  Technical  Effort  is  to  establish  plans  that  provide  the  basis  for 
scheduling,  costing,  controlling,  tracking,  and  negotiating  the  nature  and  scope  of 
the  technical  work  involved  in  system  development,  manufacturing,  use,  and 
disposal.  System  engineering  activities  must  be  integrated  into  comprehensive 
technical  planning  for  the  entire  project. 

Plan  technical  effort  involves  developing  estimates  for  the  work  to  be  performed, 
obtaining  necessary  commitments  from  interfacing  groups,  and  defining  the  plan  to 
perform  the  work. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
”  it's  enough  that  you  know  who  performs  it. 


Yes  No  N/A 

[  ]  [  ]  [  ]  Identify  resources  that  are  critical  to  the  technical  success  of  the  project. 

[  ]  [  ]  [  ]  Develop  estimates  for  the  factors  that  affect  the  magnitude  and  technical 
feasibility  of  the  project. 

[  ]  [  ]  [  ]  Develop  cost  estimates  for  all  technical  resources  required  by  the 
project. 

[  ]  [  ]  [  ]  Determine  the  technical  process  to  be  used  on  the  project. 

[  ]  [  ]  [  ]  Identify  technical  activities  for  the  entire  life  cycle  of  the  project. 

[  ]  [  ]  [  ]  Define  specific  processes  to  support  effective  interaction  with  the 
customer(s)  and  supplier(s). 

[  ]  [  ]  [  ]  Develop  technical  schedules  for  the  entire  project  life  cycle. 

[  ]  [  ]  [  ]  Establish  technical  parameters  with  thresholds  for  the  project  and  the 
system. 

[  ]  [  ]  [  ]  Use  the  information  gathered  in  planning  activities  to  develop  technical 
management  plans  that  will  serve  as  the  basis  for  tracking  the  salient 
aspects  of  the  project  and  the  systems  engineering  effort. 

[  ]  [  ]  [  ]  Review  the  technical  management  plans  with  all  ^fected  groups  and 
individuals  and  obtain  group  commitments. 

Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  12:  Plan  Technical  Effort,  Continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 
the  work  performed  as  part  of  doing  ±e  work  associated  with  this  process  area.  "Yes" 
marks  should  indicate  that  it  is  both  generally  and  visibly  practiced. 

Comments: 


ID 

Questions  relating  to  the  performance  of  the  base  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing 
this  process  area? 

2.2.2 

Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 

3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastmcture 
infrastructure  supports  Plan  Technical  Effort  by  making  the  activities  related  to  these 
^  processes  visible  to  the  organization  as  a  whole. 

Comments: 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 
supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of 
processes)  for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  12:  Plan  Technical  Effort,  Continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Plan 
management  Techical  Effort  process  area  is  managed  using  the  following  practices. 

Comments: 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

■ 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

■ 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

■ 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

■ 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

■ 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 

■ 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

1 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

1 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

4.2.2 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

1 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

■ 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

■ 

end  of  PA  12:  Plan  Technical  Effort 


PA  13:  Define  Organization's  Systems  Engineering  Process 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Define  Organization’s  Systems  Engineering  Process  is  to  create  and 
manage  the  organization's  standard  systems  engineering  processes,  which  can 
subsequently  be  tailored  by  a  project  to  form  the  unique  processes  that  it  will  follow 
in  developing  its  systems  or  products. 

Define  Organization's  Systems  Engineering  Process  involves  defining,  collecting, 
and  maintaining  the  process  that  will  meet  the  business  goals  of  the  organization,  as 
well  as  designing,  developing,  and  documenting  systems-engineering  process 
assets.  Assets  include  example  processes,  process  fragments,  process-related 
documentation,  process  architectures,  process-tailoring  rules  and  tools,  and  process 
measurements. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 

you  do  not  have  to  personally  be  involved  in  performing  the  practice 

~  it's  enough  that  you  know  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Establish  goals  for  the  organization's  systems  engineering  process  from 
the  organization's  business  goals. 

[  ]  [  ]  [  ]  Collect  and  maintain  systems  engineering  process  assets. 

[  ]  [  ]  [  ]  Develop  a  well-defined  systems  engineering  process  for  the 
organization. 

[  ]  [  ]  [  ]  Define  guidelines  for  tailoring  the  organization’s  standard  systems 

engineering  process  for  project  use  in  developing  the  project's  defined 
process. 

Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  13:  Define  Organization's  Systems  Engineering  Process, 

Continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes"  marks 

Comments-  should  indicate  that  it  is  both  generally  and  visibly  practiced. 

ID _ Questions  relating  to  the  performance  of  the  base  practices.  > 

2.2. 1  Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing 
this  process  area? 

2.2.2  Does  your  project  place  this  process  area's  work  products  under  version 

_  control  or  configuration  management  as  appropriate? 

2.3.1  Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2  Does  your  project  verify  the  compliance  of  this  process  area’s  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1  Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2  Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1  Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 


3.  _  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastructure 
infrastructure  supports  Define  Organization's  Systems  Engineering  Process  by  making  the 
Comments-  activities  related  to  these  processes  visible  to  the  organization  as  a  whole. 


ID 

Questions  relating  to  the  organization’s  infrastructure 
which  supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family 
of  processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the 
work  products  resulting  from  following  its  standard  process  (or  family 
of  processes)  for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for 
this  process  area? 

■ 

continued  on  next  page 


PA  13:  Define  Organization's  Systems  Engineering  Process, 

Continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Define 
management  Organization's  Systems  Engineering  Process  process  area  is  managed  using  the 
^  following  practices. 

Comments: 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

y/n 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

■ 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

■ 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

■ 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

■ 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

1 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

□ 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

ii 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

end  of  PA  13:  Define  Organization's  Systems  Engineering  Process 


PA  14:  Improve  Organization's  Systems  Engineering  Processes 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  pu^ose  of  Improve  Organization’s  Systems  Engineering  Processes  is  to  gain 
competitive  advantage  by  continuously  improving  the  effectiveness  and  efficiency 
of  the  systems  engineering  processes  used  by  the  organization.  It  involves 
developing  an  understanding  of  the  organization's  processes  in  the  context  of  the 
organization's  business  goals,  analyzing  the  performance  of  the  processes,  and 
explicitly  planning  and  deploying  improvements  to  those  processes. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 

you  do  not  have  to  personally  be  involved  in  performing  the  practice 

~  it's  enough  that  you  know  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Appraise  the  existing  processes  being  performed  in  the  organization  to 
understand  their  strengths  and  weaknesses. 

[  ]  [  ]  [  ]  Plaii  improvements  to  the  organization's  processes  based  on  analyzing 
the  impact  of  potential  improvements  on  achieving  the  goals  of  the 
processes. 

[]  []  []  Change  the  organization's  standard  systems  engineering  process  to 
reflect  targeted  improvements. 

[  ]  [  ]  [  ]  Communicate  process  improvements  to  existing  projects  and  to  other 
affected  groups,  as  appropriate. 

Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  14:  Improve  Organization's  Systems  Engineering 

Processes,  Continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes"  marks 

should  indicate  that  it  is  both  generally  and  visibly  practiced. 

Comments: 


ID 

Questions  relating  to  the  performance  of  the  base  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing 
this  process  area? 

2.2.2 

Does  your  project  place  this  process  area’s  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 

3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastructure 
infrastructure  supports  Improve  Organization's  Systems  Engineering  Processes  by  making 
the  activities  related  to  these  processes  visible  to  the  organization  as  a  whole. 

Comments: 


ID 

Questions  relating  to  the  organization’s  infrastructure 
which  supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family 
of  processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the 
work  products  resulting  from  following  its  standard  process  (or  family 
of  processes)  for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for 
this  process  area? 

■ 

continued  on  next  page 


PA  14:  Improve  Organization's  Systems  Engineering 
Processes,  Continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Improve 

management  Organization's  Systems  Engineering  Processes  process  area  is  managed  using  the 


Comments: 


^  following  practices. 

I  ID  Questions  relating  to  the  management  of  the  base  practices. 


2.1.1_ Does  your  project  or  organization  allocate  adequate  resources  (including 
_ people)  for  performing  this  process  area  on  your  project? 


2.1.2_ Does  your  project  or  organization  assign  responsibilities  for  developing  this 
_ process  area’s  work  products  and/or  services  for  your  project? 


2.1.3  Does  your  project  have  documented  plans,  standards,  or  procedures  in  place 
for  performing  this  process  area? 


2. 1 .4  Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 


2.1.5  Does  someone  in  the  organization  or  project  ensure  that  individuals 

performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 


2.1.6  Does  your  project  plan  the  activities  for  this  process  area? 


2  -4.1  Does  your  project  measure  the  progress  of  this  process  area  against  its  plan? 


.2  Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 


3.1.2  Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 


3.2.3  Does  your  project  apply  historical  organization  data,  standards  or  widely 

accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of  this 
process  area  ? 


4.2.1  Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 


Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process  capability? 


5.2.2  Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

5.2.3  Does  your  proj  ect  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 


tmm 

■ 


■ 

I 


end  of  PA  14:  Improve  Organization’s  Systems  Engineering  Processes 


PA  15:  Manage  Product  Line  Evolution 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Manage  Product  Line  Evolution  is  to  introduce  services,  equipment, 
and  new  technology  to  achieve  the  optimal  benefits  in  product  evolution,  cost, 
schedule,  and  performance  over  time  as  the  product  line  evolves  toward  its  ultimate 
objectives. 

An  organization  must  first  determine  the  evolution  of  a  product.  Then  the 
organization  has  to  decide  how  it  will  design  and  build  those  products  including 
critical  components,  cost-effective  tools,  and  efficient  and  effective  processes. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
—  it's  enough  that  you  know  who  performs  it. 


Yes  No 
[]  [] 
[]  [] 

[]  [] 

[]  [] 
[]  [] 


N/A 

[  ]  Define  the  types  of  products  to  be  offered. 

[  ]  Identify  new  product  technologies  or  enabling  infrastructure  that  will  help  the 
organization  acquire,  develop,  and  apply  technology  for  competitive  advantage. 

[  ]  Make  the  necessary  changes  in  the  product  development  cycle  to  support  the 
development  of  new  products. 

[  ]  Ensure  critical  components  are  available  to  support  planned  product  evolution. 

[  ]  Insert  new  technology  into  product  development,  marketing,  and  manufacturing. 


Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  15:  Manage  Product  Line  Evolution,  continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes" 

marks  should  indicate  that  it  is  both  generally  and  visibly  practiced. 


ID 

Questions  relating  to  the  performance  of  the  base  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing 
this  process  area? 

2.2.2 

Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 

3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastmcture 
infrastructure  supports  Manage  Product  Line  Evolution  by  m^ng  the  activities  related  to 
_  these  processes  visible  to  the  organization  as  a  whole. 

Comments:  ^ 


ID 

Questions  relating  to  the  organization’s  infrastructure 
which  supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family 
of  processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the 
work  products  resulting  from  following  its  standard  process  (or  family 
of  processes)  for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for 
this  process  area? 

■ 

continued  on  next  page 


PA  15:  Manage  Product  Line  Evolution,  continued 


4. 


Process 

management 


Comments: 


Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the 
Manage  Product  Line  Evolution  process  area  is  managed  using  the  following 
practices. 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

y/n 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

ii 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

■ 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

end  of  PA  15:  Manage  Product  Line  Evolution 


PA  16:  Manage  Systems  Engineering  Support  Environment 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Manage  Systems  Engineering  Support  Environment  is  to  provide 
the  technology  environment  needed  to  develop  the  product  and  perform  the  process. 
Development  and  process  technology  is  inserted  into  the  environment  with  a  goal  of 
minimizing  dismption  of  development  activities  while  upgrading  to  make  new 
technology  available. 

The  technology  needs  of  an  organization  change  over  time,  and  the  efforts 
described  in  this  process  area  must  be  re-executed  as  the  needs  evolve. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
-«  it's  enough  that  you  know  who  performs  it. 


Yes  No  N/A 

[  ]  [  ]  [  ]  Maintain  awareness  of  the  technologies  that  support  the  organization’s 
business  goals. 

[  ]  [  ]  [  ]  Determine  requirements  for  the  organization's  systems  engineering 
support  environment  based  on  organizational  needs. 

[  ]  [  ]  [  ]  Obt^n  a  systems  engineering  support  environment  that  meets  the 

requirements  established  in  Determine  Support  Requirements(above)  by 
using  the  practices  in  the  Analyze  Candidate  Solutions  process  area. 

[  ]  [  ]  [  1  Tailor  the  systems  engineering  support  environment  to  individual 
project's  needs. 


[  ]  [  ]  [  ]  Insert  new  technologies  into  the  systems  engineering  support 
environment  based  on  the  organization's  business  goals  and  the 
projects’  needs. 


[  ]  [  ]  [  ]  Maintain  the  systems  engineering  support  environment  to  continuously 
support  the  projects  dependent  on  it. 

[  ]  [  ]  [  ]  Monitor  the  systems  engineering  support  environment  for  improvement 
opportunities. 


Feel  free  to  add  clarifying  comments  in  the  margin. 


PA  16:  Manage  Systems  Engineering  Support  Environment, 

Continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes"  marks 

should  indicate  that  it  is  both  generally  and  visibly  practiced. 


ID 

Questions  relating  to  the  performance  of  the  hase  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing  this 
process  area? 

2.2.2 

Does  your  project  place  this  process  area’s  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area’s  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area’s  work  products? 

3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastructure 
infrastructure  supports  Manage  Systems  Engineering  Support  Environment  by  making  the 
activities  related  to  these  processes  visible  to  the  organization  as  a  whole. 

Comments: 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 

supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of  processes) 
for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  16:  Manage  Systems  Engineering  Support  Environment, 

Continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Manage 
management  Systems  Engineering  Support  Environment  process  area  is  managed  using  the 
_  following  practices. 

Comments:  ^  ^ 


ID  Questions  relating  to  the  management  of  the  base  practices. 


2.1.1_ Does  your  project  or  organization  allocate  adequate  resources  (including 
_ people)  for  performing  this  process  area  on  your  project? 


2.1.2_ Does  your  project  or  organization  assign  responsibilities  for  developing 
_ this  process  area's  work  products  and/or  services  for  your  project? 


2. 1 .3  Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

2.1.4  Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 


2.1.5  Does  someone  in  the  organization  or  project  ensure  that  individuals 

performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 


2.1.6  Does  your  project  plan  the  activities  for  this  process  area? 


2.4. 1  Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 


.2  Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 


3.1.2  Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 


3.2.3  Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 


4.2.1  Does  your  project  or  organization  quantitatively  determine  the  process 
_ capability  of  its  defined  process  in  this  process  area? 


.2.2  Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 


5.2.2  Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

5.2.3  Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 


■ 


I 


tmm 

■ 

I 


end  of  PA  16:  Manage  Systems  Engineering  Support  Environment 


PA  17:  Provide  Ongoing  Skills  and  Knowledge 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Provide  Ongoing  Skills  and  Knowledge  is  to  ensure  that  projects 
and  the  organization  have  the  necessary  knowledge  and  skills  to  achieve  project  and 
organizational  objectives.  To  ensure  the  effective  application  of  these  critic^ 
resources  that  are  predominantly  available  only  from  people,  the  knowledge  and 
skill  requirements  within  the  organization  need  to  be  identified,  as  well  as  the 
specific  project's  or  organization's  needs  (such  as  those  relating  to  emergent 
programs  or  technology,  and  new  products,  processes,  and  policies). 

Needed  skills  and  knowledge  can  be  provided  both  by  training  within  the 
organization  and  by  timely  acquisition  from  sources  external  to  the  organization. 
Acquisition  from  external  sources  may  include  customer  resources,  temporary 
hires,  new  hires,  consultants,  and  subcontractors.  In  addition,  knowledge  may  be 
acquired  from  subject  matter  experts. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
—  it's  enough  that  you  know  who  performs  it. 


Yes  No  N/A 


[] 

[] 

[] 

[] 

[] 

[] 

[] 

[] 


[  ]  [  ]  Identify  needed  improvements  in  skill  and  knowledge  throughout  the 

organization  using  the  projects'  needs,  organizational  strategic  plan,  and 
existing  employee  skills  as  guidance. 

[  ]  [  ]  Evaluate  and  select  the  appropriate  mode  of  acquiring  knowledge  or 
skills  with  respect  to  training  or  other  sources. 

[  ]  [  ]  Ensure  that  appropriate  skill  and  knowledge  are  available  to  the  systems 
engineering  effort. 

[  ]  [  ]  Prepare  training  materials  based  upon  the  identified  training  needs. 

[  ]  [  ]  Train  personnel  to  have  the  skills  and  knowledge  needed  to  perform 

their  assigned  roles. 

[  ]  [  ]  Assess  the  effectiveness  of  the  training  to  meet  the  identified  training 
needs. 

[  ]  [  ]  Maintain  records  of  training  and  experience. 

[  ]  [  ]  Maintain  training  materials  in  an  accessible  repository. 


-Feel  free  to  add  clarifying  comments  in  the  margin. 


PA  17:  Provide  Ongoing  Skills  and  Knowledge,  continued 

2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area.  "Yes" 

Comments-  m^ks  should  indicate  that  it  is  both  generally  and  visibly  practiced. 

ID  Questions  relating  to  the  performance  of  the  base  practices.  y/ 

2.2. 1  Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing 
this  process  area? 

2.2.2  Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1  Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2  Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1  Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2  Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1  Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 

3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastructure 

infrastructure  supports  ftovide  Ongoing  Skills  and  Knowledge  by  making  the  activities  related 
Comments-  these  processes  visible  to  the  organization  as  a  whole. 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 

supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of 
processes)  for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  17:  Provide  Ongoing  Skills  and  Knowledge,  continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the  Provide 
management  Ongoing  Skills  and  Knowledge  process  area  is  managed  using  the  following 
practices. 

Comments: 


ID 

Questions  relating  to  the  management  of  the  base  practices. 

y/n 

2.1.1 

Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 

2.1.2 

Does  your  project  or  organization  assign  responsibilities  for  developing 
this  process  area's  work  products  and/or  services  for  your  project? 

2.1.3 

Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 

2.1.4 

Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 

2.1.5 

Does  someone  in  the  organization  or  project  ensure  that  individuals 
performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 

2.1.6 

Does  your  project  plan  the  activities  for  this  process  area? 

2.4.1 

Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 

■ 

Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  that  planned? 

3.1.2 

Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 

3.2.3 

Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 

4.2.1 

Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 

1 

Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 

5.2.2 

Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 

5.2.3 

Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 

end  of  PA  17:  Provide  Ongoing  Skills  and  Knowledge 


PA  18:  Coordinate  With  Suppliers 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Coordinate  with  Suppliers  is  to  address  the  needs  of  organizations 
to  effectively  manage  the  portions  of  product  work  that  are  conducted  by  other 
organizations.  Decisions  made  as  a  part  of  this  process  area  should  be  made  in 
accordance  with  the  Analyze  Candidate  Solutions  process  area  (PAOl).  The  general 
term  supplier  is  used  to  identify  an  organization  that  develops,  manufactures,  tests, 
supports,  etc.,  a  component  of  the  system.  Suppliers  may  take  the  form  of 
vendors,  subcontractors,  partnerships,  etc.,  as  the  business  organization  warrants. 

In  addition  to  coordination  of  schedules,  processes,  and  deliveries  of  work 
products,  affected  organizations  must  have  a  shared  a  vision  of  the  working 
relationship.  Relationships  can  range  from  integrated  developer/supplier  product 
teams  to  prime  contractor/subcontractor,  to  vendors,  and  more.  A  successful 
relationship  between  an  organization  and  a  supplier  depends  on  the  capability  of 
both  organizations,  and  on  a  mutual  understanding  of  the  relationship  and 
expectations. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
—  it's  enough  that  you  know  who  performs  it. 


Yes  No  N/A 

[  ]  [  ]  [  ]  Identify  needed  system  components  or  services  that  must  be  provided 
by  other/outside  organizations. 

[  ]  [  ]  [  ]  Identify  suppliers  that  have  shown  expertise  in  the  identified  areas. 

[  ]  [  ]  [  ]  Choose  suppliers  in  accordance  with  the  Analyze  Candidate  Solutions 
process  area  (PAOl). 

[  ]  [  ]  [  ]  Provide  to  suppliers  the  needs,  expectations,  and  measures  of 

effectiveness  held  by  the  organization  for  the  system  components  or 
services  that  are  to  be  delivered. 

[]  []  []  Maintain  timely  two-way  communication  with  suppliers. 


Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 


PA  18:  Coordinate  With  Suppiiers,  continued 


2.  Performing  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  following  SE-CMM  practices  are 
the  work  performed  as  part  of  doing  the  work  associated  with  r/iw  process  area.  "Yes" 
marks  should  indicate  that  it  is  both  generally  and  visibly  practiced. 

Comments: 


ID 

Questions  relating  to  the  performance  of  the  base  practices. 

y/n 

2.2.1 

Does  your  project  follow  its  plans,  standards,  or  procedures  in  performing 
this  process  area? 

2.2.2 

Does  your  project  place  this  process  area's  work  products  under  version 
control  or  configuration  management  as  appropriate? 

2.3.1 

Does  your  project  verify  the  compliance  of  this  process  area  against  its 
applicable  plans,  standards  and/or  procedures? 

2.3.2 

Does  your  project  verify  the  compliance  of  this  process  area's  work  products 
with  its  applicable  standards  and/or  requirements? 

3.2.1 

Does  your  project  follow  a  well-defined  process  for  this  process  area? 

3.2.2 

Does  your  project  perform  defect  reviews  for  this  process  area's  work 
products? 

5.2.1 

Does  your  organization  or  project  perform  causal  analysis  of  defects  for  this 
process  area's  work  products? 

3.  Supporting  Indicate  (y,  no  or  n/a  if  not  applicable)  if  the  organizational  infrastructure  supports 
infrastructure  Coordinate  With  Suppliers  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 

Comments: 


ID 

Questions  relating  to  the  organization’s  infrastructure  which 

supports  the  projects. 

y/n 

3.1.1 

Does  your  organization  have  a  documented  standard  process  (or  family  of 
processes)  that  describes  how  to  implement  this  process  area? 

■ 

4.1.1 

Has  your  organization  established  measurable  quality  goals  for  the  work 
products  resulting  from  following  its  standard  process  (or  family  of  processes) 
for  this  process  area? 

1 

5.1.1 

Has  your  organization  established  quantitative  goals  for  improving  the 
effectiveness  of  its  standard  process  for  this  process  area? 

■ 

5.1.2 

Does  your  organization  continuously  improve  its  standard  process  for  this 
process  area? 

■ 

continued  on  next  page 


PA  18:  Coordinate  With  Suppliers,  Continued 


4.  Process  Indicate  (y,  n,  or  n/a  if  not  applicable)  if  the  process  associated  with  the 
management  Coordinate  With  Suppliers  process  area  is  managed  using  the  following 
„  practices. 

Comments:  ^ 


ID  I  Questions  relating  to  the  management  of  the  base  practices. 


2.1.1  Does  your  project  or  organization  allocate  adequate  resources  (including 
people)  for  performing  this  process  area  on  your  project? 


2. 1 .2_ Does  your  project  or  organization  assign  responsibilities  for  developing 
_ this  process  area’s  work  products  and/or  services  for  your  project? 


2.1.3  Does  your  project  have  documented  plans,  standards,  or  procedures  in 
place  for  performing  this  process  area? 


2.1.4  Does  your  project  or  organization  provide  appropriate  tools  to  support  the 
performance  of  this  process  area  on  your  project? 


2.1.5  Does  someone  in  the  organization  or  project  ensure  that  individuals 

performing  this  process  area  on  your  project  have  the  appropriate  skill  or 
knowledge? 


2.1.6  I  Does  your  project  plan  the  activities  for  this  process  area? 


2.4. 1  Does  your  project  measure  the  progress  of  this  process  area  against  its 
plan? 


Does  your  project  take  corrective  action  when  progress  varies  significantly 
from  Aat  planned? 


3.1.2  Does  your  project  tailor  the  organization’s  standard  process(es)  for  this 
process  area  to  create  a  defined  process(es)  that  meets  the  needs  of  your 
project? 


3.2.3  Does  your  project  apply  historical  organization  data,  standards  or  widely 
accepted  models  or  guidelines  to  help  plan  and  manage  the  performance  of 
this  process  area  ? 


4.2.1  Does  your  project  or  organization  quantitatively  determine  the  process 
capability  of  its  defined  process  in  this  process  area? 


.2.2  Does  your  project  take  corrective  action  as  appropriate  when  its  defined 
process  for  this  process  area  is  not  performing  within  its  process 
capability? 


5.2.2  Does  your  project  selectively  eliminate  the  causes  of  defects  in  its  defined 
process  for  this  process  area? 


5.2.3  Does  your  project  continuously  improve  its  defined  process  based  upon 
incremental  or  innovative  improvements  such  as  new  technologies? 


imm 

II 


■ 

I 


end  of  PA  18:  Coordinate  With  Suppliers 
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